Project: KICS (Provisioning / Security & Compliance)
File: README.md

The purpose of this docs is to describe KICS' E2E test suite Before running the tests, you must start the script server (NodeJS) and keep it running. Note: If you don't run this script, only the tests "031" and "052" must fail. Before running tests: ```bash cd kicsrepositoryfolder/.github/scripts/server-mock npm install npm start ``` Running E2E Tests from binary (faster) (used in dev/local): Prepare ```bash cd kicsrepositoryfolder go mod vendor ``` Build ```bash LINUX/MAC: go build -tags dev -o ./bin/kics cmd/console/main.go WINDOWS: go build -tags dev -o ./bin/kics.exe cmd/console/main.go ``` Run ```bash go test "github.com/Checkmarx/kics/v2/e2e" -v -count=1 -tags dev ``` Running E2E Tests from docker (slower) (used in CI): Prepare ```bash cd kicsrepositoryfolder go mod vendor LINUX/MAC: docker build -t kics:e2e-tests -f ./Dockerfile . WINDOWS: docker build -t kics:e2e-tests -f .\Dockerfile . ``` Run (linux/mac/bash): ```bash E2EKICSDOCKER=kics:e2e-tests go test "github.com/Checkmarx/kics/v2/e2e" -v -count=1 -tags dev ``` Run (windows) ```powershell SET "E2EKICSDOCKER=kics:e2e-tests" (or set the variable using environment variables window) go test "github.com/Checkmarx/kics/v2/e2e" -v -count=1 -tags dev ``` Test case main structure ```go var tests = []struct { name string // name of the test case args args // args structure of the test wantStatus int // expected output status code removeFiles []string // path for removing files created during the test validation Validation // custom validation function for the test case (usually regex) } ``` Test case args structure ```go type cmdArgs []string type args struct { args []cmdArgs // args to pass to kics binary expectedOut []string // path to file with expected output expectedPayload []string // path to file with expected payload expectedResult []string // path to file with expected result expectedLog LogValidation // custom validation structure } type LogValidation struct { logFile string // file name of the generated log validationFunc Validation // custom validation function to verify the log content } ``` Each test can use more or less keys within its structure, depending on the complexity of the test. A single test can contain verification based on: status code only (required for all tests) status code + custom regex verification status code + CLI output (file check) status code + payload content (file check) status code + result content (file check) status code + log content (file check with validation function) status code + result content (file check) + payload content (file check) status code + result content (file check) + payload content (file check) + log content (file check with validation function) status code + CLI output + payload content (file check) + result content (file check) + custom regex verification

<b>Note:</b> All Tests are running in parallel. To avoid errors when creating new E2E tests which contain flags that create files (payload, results, logs...) you should create this output file in a folder called "output" and the name of the generated file must contain the ID of the test case. This will guarantee that files generated by tests will not have the same name, avoiding read/write/remove errors when running several tests in parallel. The Example below (3 and 4) demonstrate E2E tests with file creation (payload/logs) <b>Example 1:</b> A test case that executes a command and checks only the status code output This type of test is simple and does not require the creation of files. ```go // E2E-CLI-020 - KICS scan with --exclude-queries flag // should not run queries that was provided in this flag. { name: "E2E-CLI-020", args: args{ args: []cmdArgs{ []string{"scan", "--exclude-queries", "15ffbacc-fa42-4f6f-a57d-2feac7365caa", "-s", "-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"}, }, }, wantStatus: []int{0}, } ``` <b>Example 2:</b> A test case that executes a command and checks: Status code output Custom validation function (regex) The custom validation function must perform a validation based on the CLI output generated by the test. The function must return true if the validation is correct and false otherwise. ```go // E2E-CLI-006 - KICS generate-id should exhibit // a valid UUID in the CLI and return exit code 0 { name: "E2E-CLI-006", args: args{ args: []cmdArgs{ []string{"generate-id"}, }, }, wantStatus: 0, // When the validation function is used, // the expectedOut function can be suppressed // The validator function allows the creation of customized functions to // perform actions on the output generated by the test. validation: func(outputText string) bool { // Usually, a validation check for elements present in the output using regex uuidRegex := "[a-f0-9]{8}-[a-f0-9]{4}-4{1}[a-f0-9]{3}-[89ab]{1}[a-f0-9]{3}-[a-f0-9]{12}" match, _ := regexp.MatchString(uuidRegex, outputText) // After the custom check, the function should return // a boolean indicating if the test has passed or failed. return match }, }, } ``` <b>Example 3:</b> A test case that executes a command and checks: Status code output Kics CLI output Generated <b>payload file</b> content The Tests that check CLI output, payload or results, need a comparison file for each output you want to compare (except for the status code). The example below contains 2 files for comparing the outputs of CLI and the payload file: <i>"E2ECLI005"</i> and <i>"E2ECLI005_PAYLOAD.json"</i>. Files used for comparing outputs must have the same name as the test and must be added to the "fixture"

In addition, it is necessary to remove the files that will be generated during the test by adding them to removeFiles (required only for test cases that generates files from results/payloads/logs). Files created by Kics during testing should always be created in the 'outputs' folder ```go // E2E-CLI-005 - KICS scan with -- payload-path flag should create a file with the // passed name containing the payload of the files scanned { name: "E2E-CLI-005", args: args{ args: []cmdArgs{ []string{"scan", "--silent", "-q", "../assets/queries", "-p", "fixtures/samples/terraform.tf", "--payload-path", "output/E2ECLI005_PAYLOAD.json"}, }, expectedOut: []string{ "E2ECLI005", }, expectedPayload: []string{ "E2ECLI005_PAYLOAD.json", }, }, wantStatus: []int{50}, removeFiles: []string{"E2ECLI005_PAYLOAD.json"}, } ``` <b>Example 4:</b> A test case that executes a command and checks: Status code output Generated <b>log file</b> content In contrast to what is expected for results and payload, log files don't have a file to compare, instead KICS validates logs like it validates CLI's output. In this case, the validation function will receive the log file data as its input and you can use any regex or do text validations in the log file content. The example below generates an output file called: <i>"E2ECLI038_LOG"</i> and creates a validation function expecting 3 matches in the generated log file content. In addition, it is necessary to remove files generated during the test, adding them to `removeFiles` (required only for test cases that generate files from results/payloads/logs). ```go // E2E-CLI-038 - KICS scan command with --log-path // should generate and save a log file for the scan { name: "E2E-CLI-038", args: args{ args: []cmdArgs{ []string{"scan", "--log-path", "output/E2ECLI038_LOG", "-q", "../assets/queries", "-p", "fixtures/samples/terraform-single.tf"}, }, expectedLog: LogValidation{ logFile: "E2ECLI038_LOG", validationFunc: func(logText string) bool { match1, _ := regexp.MatchString("Scanning with Keeping Infrastructure as Code Secure", logText) match2, _ := regexp.MatchString(`Files scanned: \d+`, logText) match3, _ := regexp.MatchString(`Queries loaded: \d+`, logText) return match1 && match2 && match3 }, }, }, wantStatus: []int{40}, removeFiles: []string{"E2ECLI038_LOG"}, } ``` E2E tests are skiped in short mode: ```go func TestE2ECLI(t *testing.T) { kicsPath := getKICSBinaryPath("") if testing.Short() { t.Skip("skipping E2E tests in short mode.") } //... } ``` <b>getKICSBinaryPath:</b> This function gets the path of the kics executable. <b>runCommand:</b> This function is intended to run kics with the input arguments sent by the test <b>readFixture & readFile:</b> These functions read a file (from its folder and name). The folders used are always: fixtures or output. <b>prepareExpected:</b> This function prepares the file to be compared, avoiding errors in reading and formatting. <b>checkLine & checkJSONLog:</b> These functions are used to check the Kics output generated in the CLI and compare it with the <b>expectedOutput</b> file. <b>fileCheck & setFields:</b> These functions read and compare the expected files with the files provided during the test. This function compares files from <b>expectedPayload</b> and <b>expectedResult</b>.