Repository: cyberark/cyberark-conjur-cli (language: python)

Issue:
CLI - Fix test env
We have the ability to run our tests as a process where we can run the CLI as a customer would, against the CLI executable where python is not required on the machine

We have started the work but it does not fully work yet.  


Discussion / hints:

Fix patch:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 8651f14c..430550bc 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -18,45 +18,169 @@ any additional information for contributing code to this project.
 
 ## Building
 
-### Egg format
+### Static/portable CLI binary
 
 ```
-$ ./bin/build
+$ ./bin/build_binary
 ```
 
-### Static/portable CLI binary
+### Egg format
 
 ```
-$ ./bin/build_binary
+$ ./bin/build
 ```
 
 ## Development
 
-- Create a directory that will hold all the virtualenv packages and files:
+To setup a development environment follow the instructions in this section. Once you have done so, you will be able to see live changes made to the CLI.
+
+1. Create a directory that will hold all the `virtualenv` packages and files. Note that you will only need to run this command once.
+
+macOS:
+
 ```
 $ python3 -m venv venv
 ```
 
-- Enable your terminal to use those files with this command:
+Windows:
+
+```
+$ py -m venv venv
+```
+
+1. Enable your terminal to use those files with this command. Note that this command will need to run each time you want to return to your virtual environment.
+
+macOS:
+
 ```
 $ source venv/bin/activate
 ```
 
-- Install requirements:
+Windows:
+
+```
+$ venv\Scripts\activate.bat
+```
+
+1. Install requirements
+
 ```
 $ pip3 install -r requirements.txt
 ```
 
-You can now run the tests and the CLI with modifiable files!
+1. You can now run the tests and the CLI with modifiable files.
+
+Check it out! Run the following prefix to start seeing changes
+
+```
+$ ./pkg_bin/conjur <command> <subcommand>
+```
 
 ## Testing
 
 ### Unit and Integration tests
 
+To run both unit and integration tests at once in a containerized environment:
+
 ```
 $ ./bin/test
 ```
 
+To run unit tests:
+```
+./bin/test_unit
+```
+
+You can run integration tests in the following ways:
+1. In a containerized environment (Python required)
+
+1. As a process (no Python required)
+
+#### Running in a containerized environment
+
+To run specific tests, perform the following:
+
+1. Drop down into a test container
+
+```
+$ ./bin/test_integration -d
+```
+
+1. Under the individual test add an identifier of your choosing
+
+For example:
+
+```
+# Example test function
+def my-integration-test()
+  ...
+
+my-integration-test.someidentifier=True
+```
+
+1. Add the identifier to the following command:
+
+```
+root@123456:/opt/conjur-api-python3# nose2 -v -X --config integration_test.cfg -A '<identifier>' $@
+
+## Example
+root@123456:/opt/conjur-api-python3# nose2 -v -X --config integration_test.cfg -A 'someidentifier' $@
+```
+
+1. You should see that only that specific test is run. Every change made locally can be seen in the container so you
+do *not* need to rebuild before running these tests again.
+
+#### Running as a process
+
+We provide the option to run the integration tests as a process. This means integration tests can run in the same way
+a user would use our CLI- without Python installed on the machine, using the Conjur CLI exec.
+
+The integration tests are wrapped in an integration_test_runner Python module to run in a Python-free environment.
+That way, tests can be run cross-platform.
+
+##### Setup
+
+1. Pack the `integration_test_runner.py` using PyInstaller in the platform to run the executable.
+
+  To pack: `pyinstaller -F test/util/test_runners/integration_test_runner.py`. Note that you will need
+  to pack each runner in each platform that you want to run the tests.
+
+1. Pack the Conjur CLI using PyInstaller in the platform to run the executable
+
+  To pack: `pyinstaller -F ./pkg_bin/conjur`. Note that you will need to pack each runner in each platform
+  that you want to run the tests. Also note that _only_ for macOS, you should pack the CLI as a directory instead of a
+  single file (using -D instead of -F). This will allow the CLI tests to run quicker.
+
+1. Run the executable: `./integration_test_runner`, supplying the below required parameters via the command line.
+
+###### Required parameters
+
+`--invoke-cli-as-process` - required to run the CLI as a process.
+
+`--cli-to-test` - path to the packed CLI executable to test against.
+
+Parameters like --url, --account, --login, --password, will used be to configure the CLI. These values are used
+before each test profile is run to configure the CLI and run the integration tests successfully.
+
+`--identifier`- the test method with this identifier will be run (`integration` by default).
+To run as a process, the identifier should be `test_with_process`.
+
+`--files-folder` - path to test assets (policy files, etc). This folder is located under `/test/test_config` in the
+repo. Copy this executable into every OS you wish to run the CLI integration tests.
+
+###### Example
+```
+./integrations_tests_runner \
+  --identifier test-with-process \
+  --url https://conjur-server \
+  --account someaccount \
+  --login somelogin \
+  --password Myp@ssw0rd1! \
+  --files-folder /test \
+  --cli-to-test /dist/conjur \
+  --invoke-cli-as-process
+```
+
 ### Linting
 
 ```
@@ -72,11 +196,72 @@ See [here](guidelines/python-cli-ux-guidelines.md) for full UX guidelines to fol
 1. Search the [open issues](../../issues) in GitHub to find out what has been planned
 2. Select an existing issue or open an issue to propose changes or fixes
 3. Add any relevant labels as you work on it
-4. Run tests as described [in the main README](https://github.com/conjurinc/conjur-api-python3#testing),
-ensuring they pass
+4. Run tests as described in the [testing section of this document](https://github.com/cyberark/conjur-api-python3/blob/master/CONTRIBUTING.md#testing), ensuring they pass
 5. Submit a pull request, linking the issue in the description
 6. Adjust labels as-needed on the issue. Ask another contributor to review and merge your code if there are delays in merging.
 
 ## Releasing
 
-TODO: Define release workflow once we have a PyPI publishing in place
+To create a tag and release follow the instructions in this section.
+
+### Update the version, changelog, and notices
+
+1. Create a new branch for the version bump
+
+1. Based on the unreleased content, determine the new version number and update the version in `version.py`
+
+1. Review the git log and ensure the changelog contains all relevant recent changes with references to GitHub issues or PRs, if possible
+
+1. Review the changes since the last tag, and if the dependencies have changed revise the [NOTICES](NOTICES.txt) file to correctly capture the added dependencies and their licenses / copyrights
+
+1. Before creating a release, ensure that all documentation that needs to be written has been written by TW, approved by PO/Engineer, and pushed to the forward-facing documentation
+
+1. Commit these changes to the branch. "Bump version to x.y.z" is an acceptable commit message and open a PR for review
+
+### Add a git tag
+
+1. Once your changes have been reviewed and merged into master, tag the version using `git tag -s v0.1.1`. Note this requires you to be able to sign releases. Consult the [github documentation on signing commits](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/managing-commit-signature-verification)
+on how to set this up. "vx.y.z" is an acceptable tag message
+1. Push the tag: `git push vx.y.z` (or `git push origin vx.y.z` if you are working from your local machine)
+
+### Add release artifacts
+
+Currently, packing the client into an executable is a manual process. For Linux and Windows, you will need to pack the client using the different VMs we have available to us. For macOS, you will need to use your local machine.
+
+#### For all OS types:
+
+1. Clone the repo by running `git clone https://github.com/cyberark/conjur-api-python3.git`
+1. Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [Python](https://realpython.com/installing-python/) if not already on the machine
+1. Activate [venv](#development) and install the requirements
+
+#### Linux
+
+1. Run `./bin/build_binary --local`. Once this is run, a `dist` folder will be created with the executable in it
+1. Once an executable has been created, run `tar -cvf conjur-cli-linux.tar conjur` to archive the file in a `tar.gz` format
+1. Add the archive file as an asset in the release
+
+#### macOS
+
+1. Run `./bin/build_binary --local`. Once this is run, a `dist` folder will be created with the executable in it
+1. Once an executable has been created, run `tar -cvf conjur-cli-darwin.tar conjur` to archive the file in a `tar.gz` format
+1. Add the archive file as an asset in the release
+
+#### Windows
+
+1. Run `pyinstaller --onefile pkg_bin/conjur`. Once this is run, a `dist` folder will be created with the executable in it
+1. Once an executable has been created, zip the executable (`zip conjur-cli-windows.zip conjur`)
+1. Add the zip as an asset in the release
+
+To copy files over from Windows VM to your local machine, use Remote Desktop redirection. In the Remote Desktop app, perform the following:
+
+1. Edit the machine and navigate to *Folders*
+1. Click on *Redirect folders* and enter the path of the shared folder. Note that you will need to establish a new connection to see the changes
+1. Drag the executable/zip to the shared folder. You should now see it on your local machine
+
+The archive files should be called the following:
+
+```
+conjur-cli-linux.tar.gz
+conjur-cli-darwin.tar.gz
+conjur-cli-windows.zip
+```
diff --git a/bin/build_binary b/bin/build_binary
index 114b5053..255502f7 100755
--- a/bin/build_binary
+++ b/bin/build_binary
@@ -4,7 +4,7 @@ BIN_BUILD_CMD="pyinstaller --onefile pkg_bin/conjur"
 
 rm -rf dist/
 
-if [[ "$1" == "-l" ]]; then
+if [[ "$1" = "--local" || "$1" = "-l" ]]; then
   $BIN_BUILD_CMD
   exit 0
 fi


Test patch:
diff --git a/bin/test_integration b/bin/test_integration
index 27cc222b..dd0c0601 100755
--- a/bin/test_integration
+++ b/bin/test_integration
@@ -5,7 +5,7 @@ cleanup() {
   docker-compose rm --stop --force -v
 }
 
-if [ "$1" == "-d" ]; then
+if [[ "$1" = "--debug" || "$1" = "-d" ]]; then
   DEBUG="true"
   shift
 fi
diff --git a/test/test_integration_configurations.py b/test/test_integration_configurations.py
index ef601bf3..92c8597c 100644
--- a/test/test_integration_configurations.py
+++ b/test/test_integration_configurations.py
@@ -50,7 +50,8 @@ def evoke_list(self, exit_code=0):
     '''
     Validates that the conjurrc was created on the machine
     '''
-    @integration_test
+
+    @integration_test(True)
     @patch('builtins.input', return_value='yes')
     def test_https_conjurrc_is_created_with_all_parameters_given(self, mock_input):
         self.setup_cli_params({})
@@ -62,7 +63,7 @@ def test_https_conjurrc_is_created_with_all_parameters_given(self, mock_input):
     '''
     Validates that the conjurrc was created on the machine when a user mistakenly supplies an extra '/' at the end of the URL
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='yes')
     def test_https_conjurrc_is_created_with_all_parameters_given(self, mock_input):
         self.setup_cli_params({})
@@ -75,7 +76,8 @@ def test_https_conjurrc_is_created_with_all_parameters_given(self, mock_input):
     Validates that if user does not trust the certificate,
     the conjurrc is not be created on the user's machine
     '''
-    @integration_test
+
+    @integration_test(True)
     def test_https_conjurrc_user_does_not_trust_cert(self):
         with patch('builtins.input', side_effect=[self.client_params.hostname, 'no']):
             self.setup_cli_params({})
@@ -89,7 +91,8 @@ def test_https_conjurrc_user_does_not_trust_cert(self):
     Validates that when the user adds the force flag,
     no confirmation is required
     '''
-    @integration_test
+
+    @integration_test(True)
     # The additional side effects here ('somesideffect') would prompt the CLI to
     # request for confirmation which would fail the test
     @patch('builtins.input', side_effect=['yes', 'somesideeffect', 'somesideeffect'])
@@ -101,7 +104,7 @@ def test_https_conjurrc_user_forces_overwrite_does_not_request_confirmation(self
 
         assert "Not overwriting" not in output
 
-    @integration_test
+    @integration_test(True)
     def test_https_cli_fails_if_cert_is_bad(self):
         # bad conjurrc
         conjurrc = ConfigFile(account=self.client_params.login, appliance_url=self.client_params.hostname,
@@ -115,7 +118,7 @@ def test_https_cli_fails_if_cert_is_bad(self):
 
         self.print_instead_of_raise_error(requests.exceptions.SSLError, "SSLError", exit_code=1)
 
-    @integration_test
+    @integration_test(True)
     def test_https_cli_fails_if_cert_is_not_provided(self):
         conjurrc = ConfigFile(account=self.client_params.login, appliance_url=self.client_params.hostname,
                               cert_file="")
diff --git a/test/test_integration_credentials.py b/test/test_integration_credentials.py
index 3c70ca8d..afabd2f0 100644
--- a/test/test_integration_credentials.py
+++ b/test/test_integration_credentials.py
@@ -65,7 +65,7 @@ def write_to_netrc(self, machine, login, password):
     3. Update their password to a randomly generated one
     4. Login with their password
     '''
-    @integration_test
+    @integration_test(True)
     @patch('builtins.input', return_value='yes')
     def test_https_netrc_is_created_with_all_parameters_given(self, mock_input):
         self.invoke_cli(self.cli_auth_params,
@@ -116,7 +116,7 @@ def test_https_netrc_is_created_with_all_parameters_given(self, mock_input):
     '''
     Validates interactively provided params create netrc
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='admin')
     def test_https_netrc_is_created_with_all_parameters_given_interactively(self, mock_pass):
         with patch('getpass.getpass', return_value=self.client_params.env_api_key):
@@ -129,7 +129,7 @@ def test_https_netrc_is_created_with_all_parameters_given_interactively(self, mo
     '''
     Validates a wrong username will raise Unauthorized error
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='somebaduser')
     def test_https_netrc_raises_error_with_wrong_user(self, mock_pass):
         with patch('getpass.getpass', return_value=self.client_params.env_api_key):
@@ -142,7 +142,7 @@ def test_https_netrc_raises_error_with_wrong_user(self, mock_pass):
     '''
     Validates a wrong password will raise Unauthorized error
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='admin')
     @patch('getpass.getpass', return_value='somewrongpass')
     def test_https_netrc_with_wrong_password(self, mock_pass, mock_input):
@@ -153,7 +153,7 @@ def test_https_netrc_with_wrong_password(self, mock_pass, mock_input):
         self.assertRegex(output, "Reason: 401")
         assert not os.path.exists(DEFAULT_NETRC_FILE)
 
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='admin')
     def test_https_netrc_is_created_when_provided_user_api_key(self, mock_pass):
         with patch('getpass.getpass', return_value=self.client_params.env_api_key):
@@ -169,7 +169,7 @@ def test_https_netrc_is_created_when_provided_user_api_key(self, mock_pass):
     Validates that when a user already logged in and reattempts and fails, the previous successful session is not removed
     '''
 
-    @integration_test
+    @integration_test(True)
     def test_https_netrc_was_not_overwritten_when_login_failed_but_already_logged_in(self):
 
         successful_run = self.invoke_cli(self.cli_auth_params,
@@ -188,7 +188,7 @@ def test_https_netrc_was_not_overwritten_when_login_failed_but_already_logged_in
     There is currently no way to fetch a host's API key so this is a work around for the 
     purposes of this test
     '''
-    @integration_test
+    @integration_test(True)
     def test_https_netrc_is_created_with_host(self):
         # Setup for fetching the API key of a host. To fetch we need to login
         self.write_to_netrc(f"{self.client_params.hostname}/authn", "admin", self.client_params.env_api_key)
@@ -224,7 +224,7 @@ def test_https_netrc_is_created_with_host(self):
     '''
     Validates when a user can logout successfully
     '''
-    @integration_test
+    @integration_test(True)
     def test_https_logout_successful(self):
 
         self.invoke_cli(self.cli_auth_params,
@@ -242,7 +242,7 @@ def test_https_logout_successful(self):
     Validates when a user attempts to logout after an already 
     successful logout, will fail
     '''
-    @integration_test
+    @integration_test(True)
     def test_https_logout_twice_returns_could_not_logout_message(self):
 
         self.invoke_cli(self.cli_auth_params,
@@ -254,12 +254,12 @@ def test_https_logout_twice_returns_could_not_logout_message(self):
         unsuccessful_logout = self.invoke_cli(self.cli_auth_params,
                                               ['logout'], exit_code=1)
 
-        self.assertEquals(unsuccessful_logout.strip(), "Failed to log out. You are already logged out.")
+        self.assertIn("Failed to log out. You are already logged out", unsuccessful_logout.strip())
         with open(DEFAULT_NETRC_FILE) as netrc_file:
             assert netrc_file.read().strip() == "", 'netrc file is not empty!'
 
 
-    @integration_test
+    @integration_test(True)
     def test_no_netrc_and_logout_returns_successful_logout_message(self):
         try:
             os.remove(DEFAULT_NETRC_FILE)
@@ -272,7 +272,7 @@ def test_no_netrc_and_logout_returns_successful_logout_message(self):
     '''
     Validates logout doesn't remove another entry not associated with Cyberark
     '''
-    @integration_test
+    @integration_test(True)
     def test_https_netrc_does_not_remove_irrelevant_entry(self):
 
         with open(f"{DEFAULT_NETRC_FILE}", "w") as netrc_test:
@@ -301,7 +301,7 @@ def test_https_netrc_does_not_remove_irrelevant_entry(self):
     Validates that when the user does not log in and attempt
     to interface with the CLI, they will be prompted to
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='someaccount')
     @patch('getpass.getpass', return_value='somepass')
     def test_user_runs_list_without_netrc_prompts_user_to_login(self, mock_pass, mock_input):
diff --git a/test/test_integration_list.py b/test/test_integration_list.py
index cd254d22..2c49884a 100644
--- a/test/test_integration_list.py
+++ b/test/test_integration_list.py
@@ -36,10 +36,10 @@ def setUp(self):
 
     # *************** TESTS ***************
 
-    @integration_test
+    @integration_test()
     def test_list_returns_resources(self):
         output = self.invoke_cli(self.cli_auth_params, ['list'])
-        self.assertEquals(f'[\n    "{self.client_params.account}:policy:root",\n'
+        self.assertIn(f'[\n    "{self.client_params.account}:policy:root",\n'
                       f'    "{self.client_params.account}:user:someuser",\n'
                       f'    "{self.client_params.account}:layer:somelayer",\n'
                       f'    "{self.client_params.account}:group:somegroup",\n'
@@ -47,12 +47,12 @@ def test_list_returns_resources(self):
                       f'    "{self.client_params.account}:variable:one/password",\n'
                       f'    "{self.client_params.account}:webservice:somewebservice"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_help_returns_help_screen(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-h'])
         self.assertIn("Name:\n  list", output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_inspect_user_returns_info_on_user(self):
         self.invoke_cli(self.cli_auth_params,
                ['policy', 'replace', '-b', 'root', '-f', self.environment.path_provider.get_policy_path("conjur")])
@@ -66,96 +66,85 @@ def test_list_inspect_user_returns_info_on_user(self):
                       f'    }}\n'
                       f']\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_kind_layer_returns_layers(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'layer'])
-            self.assertEquals(output,
-                              f'[\n    "{self.client_params.account}:layer:somelayer"\n]\n')
+            self.assertIn(f'[\n    "{self.client_params.account}:layer:somelayer"\n]\n',output)
             self.assertIn("Executing list command with the following constraints: {'kind': 'layer'}",
                           str(mock_log.output))
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_user_returns_users(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'user'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:user:someuser"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:user:someuser"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_nonexistent_returns_empty_list(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'nonexistentkind'])
-        self.assertEquals(output,
-                          f'[]\n')
+        self.assertIn(f'[]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_variable_returns_variables(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'variable'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:variable:one/password"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:variable:one/password"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_group_returns_groups(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'group'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:group:somegroup"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:group:somegroup"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_kind_policy_returns_policies(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'policy'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:policy:root"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:policy:root"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_short_host_returns_hosts(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'host'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:host:anotherhost"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:host:anotherhost"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_long_host_returns_hosts(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '--kind=host'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:host:anotherhost"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:host:anotherhost"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_kind_webservice_returns_webservices(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-k', 'webservice'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:webservice:somewebservice"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:webservice:somewebservice"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_insecure_list_prints_warning_in_log(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             self.invoke_cli(self.cli_auth_params, ['--insecure', 'list'])
             self.assertIn("Warning: Running the command with '--insecure' makes your system vulnerable to security attacks",
-                          str(mock_log.output))
+                str(mock_log.output))
 
-    @integration_test
+    @integration_test()
     def test_list_limit_short_returns_same_number_of_resources(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-l', '5'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:group:somegroup",\n'
-                          f'    "{self.client_params.account}:host:anotherhost",\n'
-                          f'    "{self.client_params.account}:layer:somelayer",\n'
-                          f'    "{self.client_params.account}:policy:root",\n'
-                          f'    "{self.client_params.account}:user:someuser"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:group:somegroup",\n'
+                      f'    "{self.client_params.account}:host:anotherhost",\n'
+                      f'    "{self.client_params.account}:layer:somelayer",\n'
+                      f'    "{self.client_params.account}:policy:root",\n'
+                      f'    "{self.client_params.account}:user:someuser"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_limit_long_returns_same_number_of_resources(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '--limit=5'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:group:somegroup",\n'
+        self.assertIn(f'[\n    "{self.client_params.account}:group:somegroup",\n'
                           f'    "{self.client_params.account}:host:anotherhost",\n'
                           f'    "{self.client_params.account}:layer:somelayer",\n'
                           f'    "{self.client_params.account}:policy:root",\n'
-                          f'    "{self.client_params.account}:user:someuser"\n]\n')
+                          f'    "{self.client_params.account}:user:someuser"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_limit_invalid_param_returns_empty_list(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-l', '0'], exit_code=1)
         self.assertIn("500 Server Error", output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_limit_string_raises_error(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-l', 'somestring'], exit_code=1)
         self.assertIn("500 Server Error", output)
@@ -163,51 +152,48 @@ def test_list_limit_string_raises_error(self):
     '''
     Validates that an invalid input (a negative number) raises an error 
     '''
-    @integration_test
+    @integration_test(True)
     def test_list_limit_invalid_negative_param_raises_error(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-l', '-5'], exit_code=1)
         self.assertIn("500 Server Error", output)
 
-    @integration_test
+    @integration_test()
     def test_list_random_input_raises_error(self):
         capture_stream = io.StringIO()
         with redirect_stderr(capture_stream):
             self.invoke_cli(self.cli_auth_params, ['list', 'someinput'], exit_code=1)
         self.assertIn("Error unrecognized arguments", capture_stream.getvalue())
 
-    @integration_test
+    @integration_test(True)
     def test_list_offset_short_returns_list_starting_at_param(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-o', '2'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:layer:somelayer",\n'
-                          f'    "{self.client_params.account}:policy:root",\n'
-                          f'    "{self.client_params.account}:user:someuser",\n'
-                          f'    "{self.client_params.account}:variable:one/password",\n'
-                          f'    "{self.client_params.account}:webservice:somewebservice"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:layer:somelayer",\n'
+                      f'    "{self.client_params.account}:policy:root",\n'
+                      f'    "{self.client_params.account}:user:someuser",\n'
+                      f'    "{self.client_params.account}:variable:one/password",\n'
+                      f'    "{self.client_params.account}:webservice:somewebservice"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_offset_long_returns_list_starting_at_param(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '--offset=2'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:layer:somelayer",\n'
-                          f'    "{self.client_params.account}:policy:root",\n'
-                          f'    "{self.client_params.account}:user:someuser",\n'
-                          f'    "{self.client_params.account}:variable:one/password",\n'
-                          f'    "{self.client_params.account}:webservice:somewebservice"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:layer:somelayer",\n'
+                      f'    "{self.client_params.account}:policy:root",\n'
+                      f'    "{self.client_params.account}:user:someuser",\n'
+                      f'    "{self.client_params.account}:variable:one/password",\n'
+                      f'    "{self.client_params.account}:webservice:somewebservice"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_offset_returns_list_of_all_resources(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-o', '0'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:group:somegroup",\n'
-                          f'    "{self.client_params.account}:host:anotherhost",\n'
-                          f'    "{self.client_params.account}:layer:somelayer",\n'
-                          f'    "{self.client_params.account}:policy:root",\n'
-                          f'    "{self.client_params.account}:user:someuser",\n'
-                          f'    "{self.client_params.account}:variable:one/password",\n'
-                          f'    "{self.client_params.account}:webservice:somewebservice"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:group:somegroup",\n'
+                      f'    "{self.client_params.account}:host:anotherhost",\n'
+                      f'    "{self.client_params.account}:layer:somelayer",\n'
+                      f'    "{self.client_params.account}:policy:root",\n'
+                      f'    "{self.client_params.account}:user:someuser",\n'
+                      f'    "{self.client_params.account}:variable:one/password",\n'
+                      f'    "{self.client_params.account}:webservice:somewebservice"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_offset_negative_raises_error(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-o', '-1'], exit_code=1)
         self.assertIn("500 Server Error", output)
@@ -215,10 +201,10 @@ def test_list_offset_negative_raises_error(self):
     # This tests is commented out because of a bug in server (https://github.com/cyberark/conjur/issues/1997)
     # where a string is considered valid input for offset. For example, when offset=somestring a list
     # of Conjur resources are returned instead of a 500 internal server error
-    # @integration_test
+    # @integration_test(True)
     # def test_list_string_offset_raises_error(self):
     #     output = self.invoke_cli(self.cli_auth_params, ['list', '-o', 'somestring'])
-    #     self.assertEquals(output,
+    #     self.assertIn(output,
     #                       f'[\n    "{self.client_params.account}:group:somegroup",\n'
     #                       f'    "{self.client_params.account}:host:anotherhost",\n'
     #                       f'    "{self.client_params.account}:layer:somelayer",\n'
@@ -227,56 +213,49 @@ def test_list_offset_negative_raises_error(self):
     #                       f'    "{self.client_params.account}:variable:one/password",\n'
     #                       f'    "{self.client_params.account}:webservice:somewebservice"\n]\n')
 
-    @integration_test
+    @integration_test(True)
     def test_list_short_search_returns_list_with_param(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-s', 'someuser'])
-        self.assertEquals(output,
-                  f'[\n    "{self.client_params.account}:user:someuser"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:user:someuser"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_long_search_returns_list_with_param(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '--search=someuser'])
-        self.assertEquals(output,
-                  f'[\n    "{self.client_params.account}:user:someuser"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:user:someuser"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_search_nonexistent_resource_returns_empty_list(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-s', 'some'])
-        self.assertEquals(output,
-                  f'[]\n')
+        self.assertIn(f'[]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_role_short_user_returns_resources_that_can_be_viewed(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-r', f'{self.client_params.account}:user:someuser'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:variable:one/password"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:variable:one/password"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_role_long_user_returns_resources_that_can_be_viewed(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', f'--role={self.client_params.account}:user:someuser'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:variable:one/password"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:variable:one/password"\n]\n', output)
 
-    @integration_test
+    @integration_test(True)
     def test_list_role_nonexistent_user_returns_forbidden(self):
         output = self.invoke_cli(self.cli_auth_params, ['list', '-r', f'{self.client_params.account}:user:nonexistinguser'], exit_code=1)
         self.assertRegex(output, '403 Client Error')
 
-    @integration_test
+    @integration_test(True)
     def test_list_combo_limit_and_kind_returns_specified_kind(self):
         self.invoke_cli(self.cli_auth_params,
                ['policy', 'load', '-b', 'root', '-f', self.environment.path_provider.get_policy_path("conjur")])
         output = self.invoke_cli(self.cli_auth_params, ['list', '-l', '2', '-k', 'host'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:host:anotherhost",\n'
-                          f'    "{self.client_params.account}:host:somehost"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:host:anotherhost",\n'
+                      f'    "{self.client_params.account}:host:somehost"\n]\n', output)
 
-    @integration_test
+    @integration_test()
     def test_list_combo_limit_and_offset_returns_specified_list(self):
         self.invoke_cli(self.cli_auth_params,
                ['policy', 'load', '-b', 'root', '-f', self.environment.path_provider.get_policy_path("conjur")])
         output = self.invoke_cli(self.cli_auth_params, ['list', '-o', '2', '-l', '3'])
-        self.assertEquals(output,
-                          f'[\n    "{self.client_params.account}:host:somehost",\n'
-                          f'    "{self.client_params.account}:layer:somelayer",\n'
-                          f'    "{self.client_params.account}:policy:root"\n]\n')
+        self.assertIn(f'[\n    "{self.client_params.account}:host:somehost",\n'
+                      f'    "{self.client_params.account}:layer:somelayer",\n'
+                      f'    "{self.client_params.account}:policy:root"\n]\n', output)
diff --git a/test/test_integration_oss.py b/test/test_integration_oss.py
index 79ee62f4..1708667f 100644
--- a/test/test_integration_oss.py
+++ b/test/test_integration_oss.py
@@ -51,7 +51,7 @@ def evoke_list(self, exit_code=0):
     OSS that doesn't have the /info endpoint
     '''
 
-    @integration_test
+    @integration_test(True)
     def test_https_conjurrc_is_created_with_no_parameters_given(self):
         with patch('builtins.input', side_effect=[self.client_params.hostname, 'yes', 'someotheraccount']):
             self.setup_cli_params({})
diff --git a/test/test_integration_policy.py b/test/test_integration_policy.py
index d307b416..544f457d 100644
--- a/test/test_integration_policy.py
+++ b/test/test_integration_policy.py
@@ -39,7 +39,7 @@ def setUp(self):
 
     # *************** TESTS ***************
 
-    @integration_test
+    @integration_test(True)
     def test_https_can_load_policy(self):
         self.setup_cli_params({})
 
@@ -49,7 +49,7 @@ def test_https_can_load_policy(self):
         for variable in variables:
             utils.assert_set_and_get(self, variable)
 
-    @integration_test
+    @integration_test()
     def test_load_policy_of_new_resources_returns_new_entry_json_data(self):
         self.setup_cli_params({})
 
@@ -82,7 +82,7 @@ def test_load_policy_of_new_resources_returns_new_entry_json_data(self):
     Validates that a policy command without a subcommand 'load'
     in this case, will fail and return help screen
     '''
-    @integration_test
+    @integration_test()
     def test_policy_load_without_subcommand_returns_help_screen(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
@@ -93,7 +93,7 @@ def test_policy_load_without_subcommand_returns_help_screen(self):
     Validates that a policy command without the policy path
     will fail and return help screen
     '''
-    @integration_test
+    @integration_test()
     def test_policy_load_without_policy_path_returns_help_screen(self):
         with redirect_stderr(self.capture_stream):
             output = self.invoke_cli(self.cli_auth_params,
@@ -103,7 +103,7 @@ def test_policy_load_without_policy_path_returns_help_screen(self):
     '''
     A non-existent policy file will return FileNotFound error message
     '''
-    @integration_test
+    @integration_test(True)
     def test_policy_load_raises_file_not_exists_error(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['policy', 'load', '-b', 'root', '-f', 'somepolicy.yml'], exit_code=1)
@@ -112,7 +112,7 @@ def test_policy_load_raises_file_not_exists_error(self):
     '''
     A policy with invalid syntax will return a Unprocessable entity error
     '''
-    @integration_test
+    @integration_test()
     def test_policy_bad_syntax_raises_error(self):
         policy = "- ! user bad syntax"
         with self.assertLogs('', level='DEBUG') as mock_log:
@@ -122,14 +122,14 @@ def test_policy_bad_syntax_raises_error(self):
         self.assertIn("422 Unprocessable Entity {\"error\":{\"code\":\"validation_failed\",\"message\":",
                   str(mock_log.output))
 
-    @integration_test
+    @integration_test()
     def test_policy_replace_load_combo_returns_help_screen(self):
         with redirect_stderr(self.capture_stream):
             output = self.invoke_cli(self.cli_auth_params,
                    ['policy', 'load', 'replace', '-b', 'root', '-f', 'somepolicy.yml'], exit_code=1)
         self.assertIn('Error unrecognized arguments: replace', self.capture_stream.getvalue())
 
-    @integration_test
+    @integration_test()
     def test_policy_insecure_prints_warning_in_log(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             self.invoke_cli(self.cli_auth_params,
@@ -138,14 +138,14 @@ def test_policy_insecure_prints_warning_in_log(self):
             self.assertIn("Warning: Running the command with '--insecure' makes your system vulnerable to security attacks",
                           str(mock_log.output))
 
-    @integration_test
+    @integration_test(True)
     def test_policy_replace_bad_syntax_raises_error(self):
         policy = "- ! user bad syntax"
         output = utils.replace_policy_from_string(self, policy, exit_code=1)
 
         self.assertIn("422 Client Error", output)
 
-    @integration_test
+    @integration_test()
     def test_https_load_policy_doesnt_break_if_no_created_roles(self):
         self.setup_cli_params({})
 
@@ -166,7 +166,7 @@ def test_https_load_policy_doesnt_break_if_no_created_roles(self):
 
         self.assertDictEqual(json_result, expected_object)
 
-    @integration_test
+    @integration_test(True)
     def test_https_can_replace_policy(self):
         self.setup_cli_params({})
 
@@ -189,7 +189,7 @@ def test_https_can_replace_policy(self):
             utils.print_instead_of_raise_error(self, old_variable, "404 Client Error: Not Found for url")
         os.remove(file_name)
 
-    @integration_test
+    @integration_test()
     def test_https_replace_policy_can_output_returned_data(self):
         self.setup_cli_params({})
 
@@ -216,7 +216,7 @@ def test_https_replace_policy_can_output_returned_data(self):
 
         self.assertDictEqual(json_result, expected_object)
 
-    @integration_test
+    @integration_test()
     def test_https_replace_policy_doesnt_break_if_no_created_roles(self):
         self.setup_cli_params({})
 
@@ -230,7 +230,7 @@ def test_https_replace_policy_doesnt_break_if_no_created_roles(self):
 
         self.assertDictEqual(json_result, expected_object)
 
-    @integration_test
+    @integration_test(True)
     def test_https_can_update_policy(self):
         self.setup_cli_params({})
 
@@ -240,7 +240,7 @@ def test_https_can_update_policy(self):
         for variable in variables:
             utils.assert_set_and_get(self, variable)
 
-    @integration_test
+    @integration_test()
     def test_https_update_policy_can_output_returned_data(self):
         self.setup_cli_params({})
 
@@ -269,7 +269,7 @@ def test_https_update_policy_can_output_returned_data(self):
 
         self.assertDictEqual(json_result, expected_object)
 
-    @integration_test
+    @integration_test()
     def test_https_update_policy_doesnt_break_if_no_created_roles(self):
         self.setup_cli_params({})
 
@@ -293,7 +293,7 @@ def test_https_update_policy_doesnt_break_if_no_created_roles(self):
     '''
     Validates that update deletes record
     '''
-    @integration_test
+    @integration_test()
     def test_policy_update_policy_removes_user(self):
         user_id = uuid.uuid4().hex
 
@@ -307,12 +307,12 @@ def test_policy_update_policy_removes_user(self):
         # Assert that user_id is not in output
         self.assertTrue(output.find(user_id) == -1)
 
-    # test_policy_update_policy_removes_user.tester=True
-    @integration_test
+
+    @integration_test()
     def test_https_can_get_whoami(self):
         self.setup_cli_params({})
 
         output = self.invoke_cli(self.cli_auth_params, ['whoami'])
         response = json.loads(output)
-        self.assertEquals(response.get('account'), f'{self.client_params.account}')
-        self.assertEquals(response.get('username'), 'admin')
+        self.assertIn(f'{self.client_params.account}', response.get('account'))
+        self.assertIn('admin', response.get('username'))
diff --git a/test/test_integration_resource.py b/test/test_integration_resource.py
index 760ddacd..874d91d4 100644
--- a/test/test_integration_resource.py
+++ b/test/test_integration_resource.py
@@ -40,7 +40,7 @@ def setUp(self):
 
     # *************** TESTS ***************
 
-    @integration_test
+    @integration_test()
     def test_resource_insecure_prints_warning_in_log(self):
         some_user_api_key = self.invoke_cli(self.cli_auth_params,
                                             ['user', 'rotate-api-key', '-i', 'someuser'])
@@ -56,7 +56,7 @@ def test_resource_insecure_prints_warning_in_log(self):
             self.assertIn("Warning: Running the command with '--insecure' makes your system vulnerable to security attacks",
                           str(mock_log.output))
 
-    @integration_test
+    @integration_test()
     def test_user_rotate_api_key_without_param_rotates_logged_in_user(self):
         some_user_api_key = self.invoke_cli(self.cli_auth_params,
                                             ['user', 'rotate-api-key', '-i', 'someuser'])
@@ -85,7 +85,7 @@ def test_user_rotate_api_key_without_param_rotates_logged_in_user(self):
 
         assert new_api_key.strip() == extract_api_key_from_message, "the API keys are not the same!"
 
-    @integration_test
+    @integration_test()
     def test_user_rotate_api_key_with_user_provided_rotates_user_api_key(self):
         some_user_api_key = self.invoke_cli(self.cli_auth_params,
                                             ['user', 'rotate-api-key', '-i', 'someuser'])
@@ -102,7 +102,7 @@ def test_user_rotate_api_key_with_user_provided_rotates_user_api_key(self):
     1. Rotate user's API key (using the admin's access token) to be able to login as them
     2. Login as that user and attempt to rotate another's API key
     '''
-    @integration_test
+    @integration_test()
     def test_unprivileged_user_cannot_rotate_anothers_api_key(self):
         unprivileged_api_key = self.invoke_cli(self.cli_auth_params,
                                                ['user', 'rotate-api-key', '-i', 'someunprivilegeduser'])
@@ -123,14 +123,14 @@ def test_unprivileged_user_cannot_rotate_anothers_api_key(self):
                                                       ['user', 'rotate-api-key', '-i', 'admin'], exit_code=1)
         self.assertIn("500 Server Error", attempt_to_rotate_admin_key)
 
-    @integration_test
+    @integration_test()
     def test_user_rotate_api_key_without_flag_returns_error(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
                             ['user', 'rotate-api-key', 'someinput'], exit_code=1)
         self.assertIn('Error unrecognized arguments: someinput', self.capture_stream.getvalue())
 
-    @integration_test
+    @integration_test()
     def test_user_rotate_api_key_flag_without_value_returns_error(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
@@ -140,9 +140,9 @@ def test_user_rotate_api_key_flag_without_value_returns_error(self):
     def test_logged_in_user_rotate_api_key_with_their_user_as_flag_returns_error(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['user', 'rotate-api-key', '-i', 'admin'])
-        self.assertEquals("Error: To rotate the API key of the currently logged-in user use this command without any flags or options", output)
+        self.assertIn("Error: To rotate the API key of the currently logged-in user use this command without any flags or options", output)
 
-    @integration_test
+    @integration_test()
     def test_user_change_password_does_not_provide_password_prompts_input(self):
         # Login as user to avoid changing admin password
         with patch('getpass.getpass', side_effect=['Mypassw0rD2\!']):
@@ -156,13 +156,13 @@ def test_user_change_password_does_not_provide_password_prompts_input(self):
                                      ['user', 'change-password'])
         self.assertIn("Successfully changed password for", output)
 
-    @integration_test
+    @integration_test(True)
     def test_user_change_password_not_complex_enough_prompts_input(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['user', 'change-password', '-p', 'someinvalidpassword'], exit_code=1)
         self.assertIn("Invalid password. The password must contain at least", output)
 
-    @integration_test
+    @integration_test()
     def test_user_change_password_meets_password_complexity(self):
         # Login as user to avoid changing admin password
         some_user_api_key = self.invoke_cli(self.cli_auth_params,
@@ -175,7 +175,7 @@ def test_user_change_password_meets_password_complexity(self):
                                  ['user', 'change-password', '-p', 'Mypassw0rD2\!'])
         self.assertIn("Successfully changed password for", output)
 
-    @integration_test
+    @integration_test()
     def test_user_change_password_empty_password_provided_prompts_input(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             with patch('getpass.getpass', side_effect=['mypassword']):
@@ -185,27 +185,27 @@ def test_user_change_password_empty_password_provided_prompts_input(self):
         self.assertIn("Invalid password. The password must contain at least", str(mock_log.output))
         self.assertIn("Invalid password. The password must contain at least", output)
 
-    @integration_test
+    @integration_test(True)
     def test_host_rotate_api_key_without_host_prompts_input(self):
         with patch('builtins.input', side_effect=['somehost']):
             output = self.invoke_cli(self.cli_auth_params,
                                      ['host', 'rotate-api-key'])
             self.assertIn("Successfully rotated API key for 'somehost'", output)
 
-    @integration_test
+    @integration_test(True)
     def test_host_rotate_api_key_with_host_provided_rotates_host_api_key(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['host', 'rotate-api-key', '-i', 'somehost'])
         self.assertIn("Successfully rotated API key for 'somehost'", output)
 
-    @integration_test
+    @integration_test()
     def test_host_rotate_api_key_without_flag_returns_error(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
                             ['host', 'rotate-api-key', 'someinput'], exit_code=1)
         self.assertIn('Error unrecognized arguments: someinput', self.capture_stream.getvalue())
 
-    @integration_test
+    @integration_test()
     def test_host_rotate_api_key_flag_without_value_returns_error(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
diff --git a/test/test_integration_variable.py b/test/test_integration_variable.py
index 10f7c3ed..e5b6d038 100644
--- a/test/test_integration_variable.py
+++ b/test/test_integration_variable.py
@@ -42,7 +42,7 @@ def setUp(self):
 
     # *************** TESTS ***************
 
-    @integration_test
+    @integration_test()
     def test_variable_get_insecure_prints_warning_in_log(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             expected_value = uuid.uuid4().hex
@@ -52,7 +52,7 @@ def test_variable_get_insecure_prints_warning_in_log(self):
             self.assertIn("Warning: Running the command with '--insecure' makes your system vulnerable to security attacks",
                           str(mock_log.output))
 
-    @integration_test
+    @integration_test()
     def test_variable_long_version_return_latest_value(self):
         """
         Note about version tests, the Conjur server only keeps a certain number of versions.
@@ -69,9 +69,9 @@ def test_variable_long_version_return_latest_value(self):
         utils.set_variable(self, variable_name, expected_value)
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', variable_name, '--version', '1'])
-        self.assertEquals(expected_value, output.strip())
+        self.assertIn(expected_value, output.strip())
 
-    @integration_test
+    @integration_test()
     def test_variable_different_version_calls_returns_different_versions(self):
         variable_name = "someversionedsecret" + uuid.uuid4().hex
         policy = f"- !variable {variable_name}"
@@ -82,15 +82,15 @@ def test_variable_different_version_calls_returns_different_versions(self):
 
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', variable_name, '--version', '1'])
-        self.assertEquals(first_version, output.strip())
+        self.assertIn(first_version, output.strip())
 
         second_version = "second_secret"
         utils.set_variable(self, variable_name, second_version)
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', variable_name, '--version', '2'])
-        self.assertEquals(second_version, output.strip())
+        self.assertIn(second_version, output.strip())
 
-    @integration_test
+    @integration_test()
     def test_variable_set_insecure_prints_warning_in_log(self):
         with self.assertLogs('', level='DEBUG') as mock_log:
             expected_value = uuid.uuid4().hex
@@ -100,27 +100,26 @@ def test_variable_set_insecure_prints_warning_in_log(self):
             self.assertIn("Warning: Running the command with '--insecure' makes your system vulnerable to security attacks",
                           str(mock_log.output))
 
-    @integration_test
+    @integration_test(True)
     def test_variable_without_subcommand_returns_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable'])
         self.assertIn("Usage:\n  conjur [global options] <command> <subcommand> [options] [args]", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test(True)
     def test_variable_short_with_help_returns_variable_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', '-h'])
         self.assertIn("Name:\n  variable", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test(True)
     def test_variable_long_with_help_returns_variable_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', '--help'])
         self.assertIn("Name:\n  variable", output)
 
-    @integration_test
+    # Note that the redirect_stderr won't work with process testing, as we redirect err into STDOUT
+    @integration_test()
     def test_variable_get_variable_without_subcommand_raises_error(self):
         with redirect_stderr(self.capture_stream):
             output = self.invoke_cli(self.cli_auth_params,
@@ -129,7 +128,7 @@ def test_variable_get_variable_without_subcommand_raises_error(self):
         self.assertIn("Error argument action: invalid choice: 'somevariable'", self.capture_stream.getvalue())
         self.assertIn("Name:\n  variable", output)
 
-    @integration_test
+    @integration_test(True)
     def test_variable_get_long_variable_returns_variable_value(self):
         expected_value = uuid.uuid4().hex
         utils.set_variable(self, 'one/password', expected_value)
@@ -137,7 +136,7 @@ def test_variable_get_long_variable_returns_variable_value(self):
                                  ['variable', 'get', '--id=one/password'])
         self.assertIn(expected_value, output)
 
-    @integration_test
+    @integration_test(True)
     def test_variable_get_short_variable_returns_variable_value(self):
         expected_value = uuid.uuid4().hex
         utils.set_variable(self, 'one/password', expected_value)
@@ -145,27 +144,27 @@ def test_variable_get_short_variable_returns_variable_value(self):
                                  ['variable', 'get', '-i', 'one/password'])
         self.assertIn(expected_value, output)
 
-    @integration_test
+    @integration_test(True)
     def test_variable_get_with_special_chars_returns_special_chars(self):
         self.invoke_cli(self.cli_auth_params,
                        ['policy', 'replace', '-b', 'root', '-f', self.environment.path_provider.get_policy_path("variable")])
         utils.set_variable(self, 'variablespecialchars', '"[]{}#@^&<>~\/''\/?\;\';\'"')
         output = utils.get_variable(self, 'variablespecialchars')
-        self.assertEquals(output.strip(), '"[]{}#@^&<>~\/''\/?\;\';\'"')
+        self.assertIn('"[]{}#@^&<>~\/''\/?\;\';\'"', output.strip())
 
-    @integration_test
+    @integration_test(True)
     def test_variable_get_variable_has_spaces_returns_variable_value(self):
         self.invoke_cli(self.cli_auth_params,
                        ['policy', 'replace', '-b', 'root', '-f', self.environment.path_provider.get_policy_path("variable")])
         utils.assert_set_and_get(self, "some variable with spaces")
 
-    @integration_test
+    @integration_test(True)
     def test_unknown_variable_raises_not_found_error(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', 'unknown'], exit_code=1)
         self.assertRegex(output, "404 Client Error: Not Found for url:")
 
-    @integration_test
+    @integration_test()
     def test_cli_can_batch_get_multiple_variables(self):
         policy, variables = utils.generate_policy_string(self)
         file_name=os.path.join(tempfile.gettempdir(), os.urandom(24).hex())
@@ -184,73 +183,68 @@ def test_cli_can_batch_get_multiple_variables(self):
         batch_result = json.loads(batch_result_string)
 
         for variable_name, variable_value in value_map.items():
-            self.assertEquals(variable_value, batch_result[variable_name])
+            self.assertIn(batch_result[variable_name], variable_value)
 
         os.remove(file_name)
 
-    @integration_test
+    @integration_test(True)
     def test_batch_existing_and_nonexistent_variable_raises_error(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', 'one/password', 'unknown'], exit_code=1)
         self.assertIn("404 Client Error", output)
 
-    @integration_test
+    @integration_test(True)
     def test_batch_existing_and_nonexistent_variable_with_spaces_raises_error(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-i', 'one/password', '"unknown password"'], exit_code=1)
         self.assertIn("404 Client Error", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test(True)
     def test_subcommand_get_short_help_returns_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '-h'])
         self.assertIn("Name:\n  get", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test(True)
     def test_subcommand_get_long_help_returns_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'get', '--help'])
         self.assertIn("Name:\n  get", output)
 
-    @integration_test
+    @integration_test()
     def test_variable_get_without_id_returns_help(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
                             ['variable', 'get'], exit_code=1)
         self.assertIn("Error the following arguments are required:", self.capture_stream.getvalue())
 
-    @integration_test
+    @integration_test()
     def test_variable_set_without_id_returns_help(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
                             ['variable', 'set'], exit_code=1)
         self.assertIn("Error the following arguments are required:", self.capture_stream.getvalue())
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test()
     def test_subcommand_set_long_help_returns_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'set', '--help'])
         self.assertIn("Name:\n  set", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test(True)
     def test_subcommand_set_short_help_returns_help(self):
         output = self.invoke_cli(self.cli_auth_params,
                                  ['variable', 'set', '-h'])
         self.assertIn("Name:\n  set", output)
 
-    # TODO This will need to be changed when UX is finalized
-    @integration_test
+    @integration_test()
     def test_subcommand_set_variable_without_value_returns_help(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
                             ['variable', 'set', '-i', 'one/password'], exit_code=1)
         self.assertIn("Error the following arguments are required:", self.capture_stream.getvalue())
 
-    @integration_test
+    @integration_test()
     def test_subcommand_set_variable_with_values_returns_help(self):
         with redirect_stderr(self.capture_stream):
             self.invoke_cli(self.cli_auth_params,
@@ -261,7 +255,7 @@ def test_subcommand_set_variable_with_values_returns_help(self):
     Validates that when the user isn't logged in and makes a request,
     they are prompted to login first and then the command is executed
     '''
-    @integration_test
+    @integration_test()
     @patch('builtins.input', return_value='admin')
     def test_variable_get_without_user_logged_in_prompts_login_and_performs_get(self, mock_input):
         # TEST_ENV is set to False so we will purposely be prompted to login
@@ -280,7 +274,7 @@ def test_variable_get_without_user_logged_in_prompts_login_and_performs_get(self
             self.assertIn('Successfully set value for variable \'one/password\'', output)
         os.environ['TEST_ENV'] = 'True'
 
-    @integration_test
+    @integration_test(True)
     def test_https_cli_can_set_and_get_a_defined_variable_if_verification_disabled(self):
         self.setup_cli_params({}, '--insecure')
         utils.assert_set_and_get(self, self.DEFINED_VARIABLE_ID)
diff --git a/test/util/test_helpers.py b/test/util/test_helpers.py
index 0c76b516..37e0f9ca 100644
--- a/test/util/test_helpers.py
+++ b/test/util/test_helpers.py
@@ -1,4 +1,5 @@
 import os
+import signal
 import tempfile
 import uuid
 from unittest.mock import patch
@@ -8,6 +9,18 @@ def remove_file(file_path):
     if os.path.isfile(file_path):
         os.remove(file_path)
 
+def run_func_with_timeout(timeout, func,*args):
+    # IMPORTANT! this must raise an exception for the interrupt to work with blocking functions
+    def interrupted(signum, frame):
+        raise RuntimeError("interrupted")
+    signal.signal(signal.SIGALRM, interrupted)
+    # set alarm
+    signal.alarm(timeout)
+    function_output = func(*args)
+    # disable the alarm after success
+    signal.alarm(0)
+    return function_output
+
 @patch('builtins.input', return_value='yes')
 def init_to_cli(self, mock_input):
     self.invoke_cli(self.cli_auth_params,
@@ -36,7 +49,9 @@ def assert_set_and_get(self, variable_id):
 
     set_variable(self, variable_id, expected_value)
     output = get_variable(self, variable_id)
-    self.assertEquals(expected_value, output.strip())
+    # using AssertIn and not AssertEqual as in the process we get the entire Conjur STDOUT stdout
+    self.assertIn(expected_value, output.strip())
+
 
 def assert_variable_set_fails(self, variable_id, error_class, exit_code=0):
     with self.assertRaises(error_class):
@@ -83,7 +98,6 @@ def load_policy(self, policy_path, exit_code=0):
                            ['policy', 'load', '-b', 'root', '-f', policy_path], exit_code=exit_code)
 
 def load_policy_from_string(self, policy, exit_code=0):
-    output = None
     file_name = os.path.join(tempfile.gettempdir(), os.urandom(24).hex())
     with open(file_name, 'w+b') as temp_policy_file:
         temp_policy_file.write(policy.encode('utf-8'))
diff --git a/test/util/test_infrastructure.py b/test/util/test_infrastructure.py
index ebbd73f7..394a514d 100644
--- a/test/util/test_infrastructure.py
+++ b/test/util/test_infrastructure.py
@@ -11,15 +11,19 @@
 from conjur.cli import Cli
 
 
-def integration_test(original_function):
-    @wraps(original_function)
-    def test_wrapper_func(self, *inner_args, **inner_kwargs):
-        return original_function(self, *inner_args, **inner_kwargs)
-
-    test_wrapper_func.integration = True
+def integration_test(should_run_as_process=False):
+    def function_decorator(original_function):
+        @wraps(original_function)
+        def test_wrapper_func(self, *inner_args, **inner_kwargs):
+            return original_function(self, *inner_args, **inner_kwargs)
 
-    return test_wrapper_func
+        # supports tests running as a process
+        if should_run_as_process:
+            test_wrapper_func.test_with_process = True
 
+        test_wrapper_func.integration = True
+        return test_wrapper_func
+    return function_decorator
 
 def cli_test(cli_args=[], integration=False, get_many_output=None, get_output=None, list_output=None,
              policy_change_output={}, whoami_output={}, rotate_api_key_output={}):
diff --git a/test/util/test_runners/integration_test_case.py b/test/util/test_runners/integration_test_case.py
index 0e17f103..d530dc84 100644
--- a/test/util/test_runners/integration_test_case.py
+++ b/test/util/test_runners/integration_test_case.py
@@ -16,8 +16,9 @@
 # Internals
 from conjur.cli import Cli
 from test.util.test_runners.params import ClientParams, TestEnvironmentParams
+from test.util import test_helpers as utils
 
-
+MAX_INTERACTIONS_ALLOWED = 5
 class IntegrationTestCaseBase(TestCase):
 
     def __init__(self, testname, client_params: ClientParams = None,
@@ -65,13 +66,23 @@ def invoke_cli_as_code(test_runner, *args, exit_code=0):
                             f"ERROR: CLI returned an unexpected error status code: '{cli_args}'. Output: {capture_stream.getvalue()}")
     return capture_stream.getvalue()
 
-
 def invoke_cli_as_process(test_runner, *args, exit_code=0) -> str:
     """
-    Invoke cli command using cli executable.
-    The usecase of this is when we run the tests
-    on tests environment. will help tests integration
-    on Windows, macOS, RHEL
+    Invoke cli command using CLI executable. This flow is used for when
+    we run our tests in a test environment and will help run our integration
+    tests on Windows, macOS, RHEL.
+    This function raises a Conjur CLI process with test arguments supplied
+    by the user. These inputs are passed to the Conjur process using the
+    process.stdin.write method. Interactive input is provided by
+    "unittest.patch" module defined in the integration tests code. This
+    is not a straightforward operation. Notice two arguments below:
+    1) max_interactions - > we limit the number of interactive command
+    as the "patch" module can sometimes interact infinite number of
+    times example:
+        @patch('builtins.input', return_value='yes')
+    2) interactive_input - raise a prompt with a timeout to initiate
+    interaction with "patch" and retrieve the desired value. Note that
+    the timeout is important in case no input will come from "patch".
     @param test_runner:
     @param args: the cli args input
     @param exit_code:
@@ -79,11 +90,41 @@ def invoke_cli_as_process(test_runner, *args, exit_code=0) -> str:
     """
     cli_args = list(itertools.chain(*args))
     run_cli_cmd = f"{test_runner.environment.cli_to_test}"
-
-    child = sp.Popen([run_cli_cmd] + cli_args, stdout=sp.PIPE)
-    output = child.communicate()[0]
-    process_exit_code = child.returncode
-
-    test_runner.assertEqual(process_exit_code, exit_code,
-                            "ERROR: CLI returned an unexpected error status code: '{}'".format(cli_args))
+    with sp.Popen([run_cli_cmd] + cli_args, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.STDOUT) as process:
+        try:
+            # number of max interactions allowed
+            max_interactions = MAX_INTERACTIONS_ALLOWED
+            while process.poll() is None and max_interactions > 0:
+                max_interactions -= 1
+                # timeout must be integer and should be the maximum seconds
+                # waiting for the conjurCli to process an input
+                interactive_input = get_input_if_exist(timeout=1)
+                if interactive_input:
+                    # pass the interactive input into conjurCli process
+                    process.stdin.write(interactive_input)
+                else:
+                    break
+        except Exception as e:
+            print(e)
+        output = process.communicate(timeout=30)[0]
+        process_exit_code = process.returncode
+        test_runner.assertEqual(process_exit_code, exit_code,
+                                "ERROR: CLI returned an unexpected error status code: '{}'".format(cli_args))
     return output.decode('utf-8')
+
+def get_input_if_exist(timeout=0.1):
+    # get the input from the "unittest.patch"
+    def get_input():
+        try:
+            data = input()
+            return data
+        except:
+            # timeout
+            return
+    try:
+        sys_in = utils.run_func_with_timeout(timeout, get_input)
+        if sys_in:
+            return (sys_in + "\n").encode('utf-8')
+        return None
+    except:
+        return None
diff --git a/test/util/test_runners/integrations_tests_runner.py b/test/util/test_runners/integrations_tests_runner.py
index 1215e2eb..6a468c9f 100644
--- a/test/util/test_runners/integrations_tests_runner.py
+++ b/test/util/test_runners/integrations_tests_runner.py
@@ -2,10 +2,6 @@
 This is a runner module for all integration tests.
 The purpose of this file is to be a cross platform tests runner
 for all integration tests
-
-Roadmap:
-1) run all integration tests from this file
-2) set up the system env to run the tests
 """
 
 # Builtins
@@ -14,16 +10,19 @@
 sys.path.append('.')
 sys.path.append('..')
 # Third party
-import uuid
 import unittest.mock
 import unittest
 import base64
 import requests
-from pathlib import Path
 import logging
 import subprocess as sb
 import shutil
 
+# NOTE do not delete this import even though it seems to not be used. This used to
+# run the integration tests as a process. Otherwise this dependency will not be available
+# once the test module is packed
+import uuid
+
 # Internals
 from test.test_integration_policy import CliIntegrationPolicy
 from test.test_integration_variable import CliIntegrationTestVariable
@@ -90,7 +89,7 @@ def start_env(self):
     def __create_conjurrc_file(self):
         path_provider = TestRunnerPathProvider.getInstance()
         proc = sb.Popen(
-            [self.runner_args.working_cli_path_only_for_init, "init", "-a", f"{self.runner_args.account}", "-u",
+            [self.runner_args.cli_to_test, "init", "-a", f"{self.runner_args.account}", "-u",
              f"{self.runner_args.hostname}",
              "--force"],
             stdin=sb.PIPE,
@@ -105,7 +104,7 @@ def __create_conjurrc_file(self):
     def __create_netrc_file(self):
         path_provider = TestRunnerPathProvider.getInstance()
         proc = sb.Popen(
-            [self.runner_args.working_cli_path_only_for_init, "login", "-i", f"{self.runner_args.login}", "-p",
+            [self.runner_args.cli_to_test, "login", "-i", f"{self.runner_args.login}", "-p",
              f"{self.runner_args.password}"],
             stdin=sb.PIPE,
             stdout=sb.PIPE,
@@ -119,7 +118,8 @@ def __create_netrc_file(self):
 
     def __get_api_key(self):
         url = f"{self.runner_args.hostname}/authn/{self.runner_args.account}/login"
-        encode = base64.b64encode(f"admin:{self.runner_args.password}".encode('utf-8')).decode('utf-8')
+        encode = base64.b64encode(f"{self.runner_args.login}:{self.runner_args.password}".encode('utf-8')).decode(
+            'utf-8')
         headers = {'Authorization': f'Basic {encode}'}
 
         response = requests.request("GET", url, headers=headers, data={},
@@ -139,6 +139,7 @@ def __run_test_cases(self, test_cases_class: IntegrationTestCaseBase):
             specific_test = test_cases_class(test_name, self.client_params, self.env_params)
             suite.addTest(specific_test)
         result = unittest.TextTestRunner().run(suite)
+        print("Failed: ","\n".join([str(f[0]).split(" ")[0] for f in result.failures]))
         return result.wasSuccessful()
 
     def __get_relevant_tests_for_class(self, class_name: IntegrationTestCaseBase) -> list:
diff --git a/test/util/test_runners/params.py b/test/util/test_runners/params.py
index 5c0e920b..f4f0c3a7 100644
--- a/test/util/test_runners/params.py
+++ b/test/util/test_runners/params.py
@@ -24,10 +24,10 @@ class TestEnvironmentParams:
     Default params are used for test_integration script
     """
 
-    def __init__(self, cli_to_test_path=None, invoke_process=False,
+    def __init__(self, cli_to_test=None, invoke_process=False,
                  path_provider=None):
         self.invoke_process = invoke_process
-        self.cli_to_test_path = cli_to_test_path
+        self.cli_to_test = cli_to_test
         if path_provider:
             self.path_provider = path_provider
         elif TestRunnerPathProvider.getInstance():
diff --git a/test/util/test_runners/test_runner_args.py b/test/util/test_runners/test_runner_args.py
index 13630339..81e2933a 100644
--- a/test/util/test_runners/test_runner_args.py
+++ b/test/util/test_runners/test_runner_args.py
@@ -1,5 +1,18 @@
 from argparse import ArgumentParser
 
+# *************** DEVELOPER NOTE ***************
+# We have the ability to run our tests as a process which means we can run our tests as a user would,
+# without Python on the machine, using the Conjur CLI exec a customer would. Currently, some tests still
+# cannot run as a process. Tests that meet the following criteria cannot be run as a process and are given
+# the integration_test() decorator:
+# 1. any test that contains of error/stderr
+# 2. any test that runs a command that uses getpass (login and user commands)
+# 3. any test that has assertEquals. All new tests should use assertIn
+# 4. any test that uses a tmp file (policy command)
+# 5. any test that uses contents returned from invoke_cli
+#
+# The main reason for these limitations is that we run in debug mode and all items that are printed to the
+# screen are captured.
 
 class TestRunnerArgs:
     """
@@ -7,8 +20,7 @@ class TestRunnerArgs:
     """
 
     def __init__(self, run_oss_tests, url, account, login, password, invoke_cli_as_process,
-                 working_cli_path_only_for_init: str,
-                 cli_to_test: str, files_folder: str, allow_same_binaries=False,
+                 cli_to_test: str, files_folder: str,
                  test_name_identifier="integration"):
         self.run_oss_tests = run_oss_tests
         self.hostname = url
@@ -16,37 +28,37 @@ def __init__(self, run_oss_tests, url, account, login, password, invoke_cli_as_p
         self.login = login
         self.password = password
         self.test_name_identifier = test_name_identifier
+        # Tests
+        if invoke_cli_as_process and test_name_identifier == 'integration':
+            self.test_name_identifier = "test_with_process"
         self.invoke_cli_as_process = invoke_cli_as_process
-        self.working_cli_path_only_for_init = working_cli_path_only_for_init
         self.cli_to_test = cli_to_test
         self.files_folder = files_folder
-        # will throw if the user test the same known working binaries
-        if cli_to_test is not None and not allow_same_binaries and working_cli_path_only_for_init.lower() == cli_to_test.lower():
-            raise RuntimeError("cli to start enviroment and tested cli are on the same path")
 
     @staticmethod
     def create_from_args():
         """
-        --invoke_cli_as_process , required to run the CLI as a process
-        --working_cli_path the path to a working CLI executable.
-        This path is needed for initial configuration of the CLI.
-        You can find this executable under /test/test_config/binaries/config
-        in the repo. Copy this executable into every OS you wish to run the
-        CLI integration tests.
-        Parameters like -u, -a, -l, -p, will configure the conjurrc and netrc
-        needed to run the integration tests successfully.
-        --cli_to_test path to the CLI executable to test against.
-        --files_folder path to test assets (policy files, etc).
+        --invoke-cli-as-process, required to run the CLI as a process
+
+        Parameters like --url, --account, --login, --password, will used be to configure the CLI. These values
+        are used before each test profile is run to configure the CLI and run the
+        integration tests successfully
+
+        --cli-to-test, path to the packed CLI executable to test against.
+
+        --files-folder path to test assets (policy files, etc).
         You can find this folder under /test/test_config in the repo.
         Copy this executable into every OS you wish to run the CLI integration tests.
-        --identifier the test method with this identifier will be run (integration by default).
+
+        --identifier the test method with this identifier will be run (`integration` by default).
+        To run as a process, the identifier should be `test_with_process`
         """
         parser = ArgumentParser(description='Test arguments')
 
         # Add the arguments
         parser.add_argument('--oss', dest='run_oss_tests', action='store_true',
                             help='Added to run OSS-specific tests')
-        parser.add_argument('-p_invoke', '--invoke_cli_as_process', action='store_true',
+        parser.add_argument('-p-invoke', '--invoke-cli-as-process', action='store_true',
                             help='If added, integration tests will run as a process executable. Otherwise it will run '
                                  'as code, requiring Python')
         parser.add_argument('-i', '--identifier', dest='test_name_identifier', action='store', default='integration',
@@ -56,13 +68,9 @@ def create_from_args():
         parser.add_argument('-a', '--account', dest='account', action='store', default='dev', help='account name')
         parser.add_argument('-l', '--login', dest='login', action='store', default='admin', help='user name')
         parser.add_argument('-p', '--password', dest='password', action='store', help='the user password')
-        parser.add_argument('-w', '--working_cli', dest='working_cli_path_only_for_init', action='store',
-                            help='the cli path using to set up the enviorment')
-        parser.add_argument('-c', '--cli_to_test', dest='cli_to_test', action='store',
+        parser.add_argument('-c', '--cli-to-test', dest='cli_to_test', action='store',
                             help='the cli binaries to test')
-        parser.add_argument('-f', '--files_folder', dest='files_folder', action='store', default='./test',
+        parser.add_argument('-f', '--files-folder', dest='files_folder', action='store', default='./test',
                             help='where the test assets are located')
-        parser.add_argument('-s', '--same', dest='allow_same_binaries', action='store_true',
-                            help='If added cli_to_test_path and working_cli can address the same path')
         args = parser.parse_args()
         return TestRunnerArgs(**vars(args))