Tutorial: Getting started with Microsoft RESTler
By: Pamella Bezerra
Tags: Restler, Fuzzing, Rest Api, Tutorial, Testing Tools

Pre-requisites

Install Python 3.8.2 and .NET core SDK 3.1, according to your appropriate OS.

Set up RESTler

Create a directory to place the RESTler binaries:

mkdir C:\restler_bin

Clone the RESTler repository, switch to its root directory and then run the following Python script:

python ./build-restler.py --dest_dir C:\restler_bin

Once it is over, four folders will have been created (compiler, engine, restler, and resultsAnalyzer).

Time to play

📑 Disclaimer: For this tutorial, we are going to use the PokeAPI as our example REST API . As it is a real API and not a fake or demo one, it servers over 60,000,000 API calls each month, the execution of the last mode, Fuzz, will be skipped because it is the most aggressive one and may create outages in the service under test.

RESTler runs in 4 main ordered modes and we will get through all of them now.

1. Compile

In this first step, the tool analyzes the Swagger/OpenAPI spec and generates a fuzzing grammar, which contains information about parameters and responses for individual requests and the dependencies between requests.

To begin let’s create a new directory where will run RESTler on our example.

mkdir C:\ restler-test cd C:\ restler-test

Then download the pokeapi.yaml and save it to C:\restler-test so we can compile it with RESTler.

C:\restler_bin\restler\Restler.exe compile --api_spec C:\restler-test\pokeapi.yaml

The compile command creates a new subdirectory of the same name (Compile) where the results of the compilation are stored in several files, such as grammar.py , grammar.json , dict.json , engine_settings.json , and config.json .

2. Test

Now that we have the compiled grammar, the tool will attempt to execute successfully all request types of the Swagger specification at least once in order to identify request types that are unreachable/exercisable with the current test setup.

C:\restler_bin\restler\Restler.exe test --gr

ammar_file C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

This step may take a while but once it is done the results are stored in a new subdirectory of the same name (Test). Looking at the subdirectory C:\restler-test\Test\RestlerResults\experiment<...>\logs , there will be a file named main.txt which lists one-by-one all the API requests in the Swagger spec and whether the request has been successfully executed ( VALID *) or not ( INVALID ), and also reveals the total spec coverage at the bottom of that file.

*In this context, a valid request means RESTler executed the request and also get a 20x HTTP status code as a response.

Here is the main.txt output I got for the PokeAPI grammar:

2020-11-21 22:25:50.269: Going to fuzz a set with 2 requests 2020-11-21 22:25:50.269: Request-0: Value Combinations: 1 2020-11-21 22:25:50.269: Request-1: Value Combinations: 2 2020-11-21 22:25:50.269: Avg. Value Combinations per Request: 1.5 2020-11-21 22:25:50.269: Median Value Combinations per Request: 1.5 2020-11-21 22:25:50.269: Min Value Combinations per Request: 1 2020-11-21 22:25:50.269: Max Value Combinations per Request: 2 2020-11-21 22:25:50.269: Total dependencies: 0 2020-11-21 22:25:50.271: Starting directed-smoke-test 2020-11-21 22:25:50.288: Will attempt to render 2 requests found Rendering request 0 from scratch 2020-11-21 22:31:51.911: Request 0 2020-11-21 22:31:51.912: Endpoint - /pokemon/{idOrName} 2020-11-21 22:31:51.912: Hex Def - 68da971679008d9273ce4bbcb33f7faa1522c900 2020-11-21 22:31:51.912: Sequence length that satisfies dependencies: 1 2020-11-21 22:31:51.932: Rendering INVALID 2020-11-21 22:31:51.951: A bug code was received after rendering this request. - restler_static_string: 'GET ' - restler_static_string: '/' - restler_static_string: 'api' - restler_static_string: '/' - restler_static_string: 'v2' - restler_static_string: '/' - restler_static_string: 'pokemon' - restler_static_string: '/' + restler_fuzzable_int: ['0', '1'] - restler_static_string: ' HTTP/1.1\r

' - restler_static_string: 'Accept: application/json\r

' - restler_static_string: 'Host: pokeapi.co\r

' - restler_static_string: '\r

' Rendering request 1 from scratch 2020-11-21 22:35:53.090: Request 1 2020-11-21 22:35:53.090: Endpoint - /pokemon 2020-11-21 22:35:53.090: Hex Def - b093ad80f75a0b14181970c3262f13d41f570641 2020-11-21 22:35:53.090: Sequence length that satisfies dependencies: 1 2020-11-21 22:35:53.113: Rendering INVALID 2020-11-21 22:35:53.135: A bug code was received after rendering this request. - restler_static_string: 'GET ' - restler_static_string: '/' - restler_static_string: 'api' - restler_static_string: '/' - restler_static_string: 'v2' - restler_static_string: '/' - restler_static_string: 'pokemon' - restler_static_string: ' HTTP/1.1\r

' - restler_static_string: 'Accept: application/json\r

' - restler_static_string: 'Host: pokeapi.co\r

' - restler_static_string: '\r

' 2020-11-21 22:35:53.481: Final Swagger spec coverage: 0 / 2 2020-11-21 22:35:53.481: Rendered requests: 2 / 2 2020-11-21 22:35:53.481: Rendered requests with "valid" status codes: 0 / 2 2020-11-21 22:35:53.481: Num fully valid requests (no resource creation failures): 0 2020-11-21 22:35:53.481: Num requests not rendered due to invalid sequence re-renders: 0 2020-11-21 22:35:53.481: Num invalid requests caused by failed resource creations: 0 2020-11-21 22:35:53.481: Total Creations of Dyn Objects: 0 2020-11-21 22:35:53.481: Total Requests Sent: {'gc': 0, 'main_driver': 3} 2020-11-21 22:35:53.481: Bug Buckets: {'main_driver_timeout': 2}

3. Fuzz-lean

In Fuzz-lean mode, RESTler executes once every <endpoint, method> in the compiled RESTler PokeAPI grammar with a default set of checkers looking for bugs that can be found quickly.

C:\restler_bin\restler\Restler.exe fuzz-lean --gr

ammar_file C:\restler-test\Compile\grammar.py --dictionary_file C:\restler-test\Compile\dict.json --settings C:\restler-test\Compile\engine_settings.json --no_ssl

This step may also take time to finish. Likewise the past modes, the results are stored in a new subdirectory of the same name (FuzzLean) and at the C:\restler-test\FuzzLean\RestlerResults\experiment<...>\logs\ subdirectory you will find the main.txt file with the same coverage results as the previous Test run. However, in the Fuzz-lean mode, a new bug_buckets directory is created containing four files but for now you can pay attention to bug_buckets.txt which includes a list of each unique sequence that found a bug*.

*There are a few types of bug that RESTler is able to detect, see more details on that matter here.

Here is the .txt output I got for the PokeAPI grammar:

main_driver_timeout: 2

Total Buckets: 2

-------------

main_driver_timeout - Bug was reproduced - main_driver_timeout_1.txt

Hash: main_driver_timeout_466d56fb305ae37cfc6d92596aeb99ca258d6d54

GET /api/v2/pokemon/1 HTTP/1.1\r

Accept: application/json\r

Host: pokeapi.co\r

\r



--------------------------------------------------------------------------------

main_driver_timeout - Bug was reproduced - main_driver_timeout_2.txt

Hash: main_driver_timeout_2b385ce87900b8609cb646d75fd7cd963584586a

GET /api/v2/pokemon HTTP/1.1\r

Accept: application/json\r

Host: pokeapi.co\r

\r



--------------------------------------------------------------------------------

4. Fuzz

In the last step, RESTler will fuzz the service under test for a longer period of time with the purpose of catching even more bugs and issues (resource leaks, perf degradation, backend corruptions, etc.).

⚠️The Fuzz mode is the more aggressive and may create outages in the service under test if the service is poorly implemented.

📑 Disclaimer: As I had mentioned before, the Fuzz mode was not executed with the PokeAPI because it is an authentic service and our aim to use it was to show how the tool handles a real API and not to provoke outages to external applications.

In order to not encourage you to run this mode with a service that you do not have permission to do so, we will not write the command here but its syntax is available in the RESTler docs.