Package: change-analyzer
Change analyzerChange analyzerAboutMain Data featuresData CollectionData ReconstructionData AnalysisData ValidationSetup and usage in Windows environment1. Download required 3rd party softwares:2. Enable developer mode in Windows3. Create virtual environment4. Create configuration file5. Run the software6. Compare resultsAvailable commands1. ca-run2. ca-compareConfiguration filesAcknowledgementAboutChange-Analyzer (CA in short) is a framework built utilizing ML techniques, leveragingOpenAI Gym libraryandLudwigAI framework.
CA allows product teams to get feedback regarding their software product, aka SUT (System Under Test).Main Data featuresEssentially, is built around the following main Data features:Data Collection, done through Automated Exploratory TestingData Reconstruction, done through Automated Regression TestingData Analysis, done through Change DetectionData Validation, done through Verdict GenerationData CollectionAfter installation and configuration (see sections below) Data Collection can be started.
An agent is exploring the SUT without prior knowledge of it, recording testing sequence and state of the application.
Once the execution is completed, the Data is saved in a csv file, available in a dedicated folder, from recordings.
Currently, the Data consists of the following information regarding the steps of the executed sequence:Timestamp: an Integer with the execution time in epochSequenceId: a String with a unique ID to identify the sequencePageSourceBefore: a String with the page source (before step execution)PageSourceAfter: a String with the page source (after step execution)ActionToPerform: a String with the action that will be performed nextActionImageBefore: a numpy.ndarray with the image of SUT (before step execution)ActionImageAfter: a numpy.ndarray with the image of SUT (after step execution)Data ReconstructionOnce several sequences exist, they can be used for Data Reconstruction.
An agent is rerunning the specified existing sequence, recording a new state of the application.
Data collected has the same structure as described in the above section.Data AnalysisIn this phase, two sequences are analyzed. The sequences are the same, from steps perspective, but are executed against
different SUT versions (deltas).A report is created, to showcase the differences between the two sequences, if any. The goal is to provide a visual tool
for change detection.The report will contain the following info:The info regarding which is the expected sequence and which is the actual sequenceA visual indicator for each step if it was considered passed or failedA description of the executed steps stepA short info to support the step statusExpected image of the SUT, from expected sequenceActual image of the SUT, from actual sequenceActual and expected images may contain highlighted elements, if they were detected to be part of a changeData ValidationUsing ML techniques, in this phase, the tool should be able to say if the detected changes are expected behavior of faults in the SUT.
Data Validation will be cover in more details later, because currently is in planning phase.Setup and usage in Windows environment1. Download required 3rd party softwares:DownloadSelenium standalone server.Version 3.141.59 confirmed as working.Put .jar file to place where you can easily find it.FOR DESKTOP APP TESTING: DownloadWindows Application Driver.Version 1.2.1 confirmed as working.Install downloaded .msi file.FOR WEBSITE TESTING: DownloadChrome driver.Version 94.0.4606.61 confirmed as working.Put binary file to place where you can easily find it.2. Enable developer mode in WindowsWindows logo -> developer settings -> EnableOfficial documentationif needed.3. Create virtual environment3.1. Virtualenv optionRequires Python 3.7 or newer installed and added to PATH variable.Runinstall.batlocated in source code root directory.This script will install necessary dependencies defined in setup.py.3.2. Conda optionIf preferred, it is also possible to use a Conda environment.Open an Anaconda terminal and go to project's main folderRunconda env create -f requirements.ymlThe above command will create the conda environmentchange_analyzerincluding all the needed dependenciesIn order to use the Conda environment, you can either attach it to a project within your favorite IDE, or activate
it on command line withconda activate change_analyzer4. Create configuration fileIt's mandatory to tell change-analyzer how to execute testing. For this reason .ini configuration is needed. Create configure now before running the software.More detailscan be found in later part of this README.5. Run the softwareRequires Java 8 installed and added to PATH variable.Start Selenium standalone server:Open CMD and change directory to Selenium directory (downloaded in step 1).java -jar selenium-server-standalone-[version].jarVersion needs to be change accordingly.Minimize CMD and let it run on background.FOR DESKTOP APP TESTING: Start Windows Application DriverRunWinAppDriver.exeDefault location C:\Program Files (x86)\Windows Application DriverMinimize CMD and let it run on background.FOR WEBSITE TESTING:Runchromedriver.exeMinimize CMD and let it run on background.Run the software:Open CMD and change directory to source code root.Activate virtual environment:.venv\Scripts\activateGenerate new report:ca-run --config [configuration filename].iniResults appear inrecordingsdirectory.6. Compare resultsRequires two generated test results.Open CMD and change directory to source code root.Activate virtual environment:.venv\Scripts\activateCompare results:ca-compare --sequence1_folder [Test folder 1] --sequence2_folder [Test folder 2]Replace[Test folder 1]with real results folder name (example 2021_10_05-14_15_00).Test results are generated torecordingsdirectory.Argument paths starts fromrecordingsdirectory.Available commands1. ca-runIt is used to explore a given application and generate new test results.Parameters--config(mandatory) - Defines which .ini file to use. Currently, Windows and Web platforms are supported.--csv_folder(optional) - it triggers the Replay mode. Replay will run the sequence found within the csv_folder.--steps(optional) - it represents the number of total steps desired for the sequence to have (default is 10)--strategy(optional) - it defines the agent strategy, either random or rl (reinforcement learning)--model_dir(optional) - it specifies the name of the pretrained model to use from pretrained_models folderResultsResults can be found withinrecordingsfolder in a dedicated sub-folder.Command examples1a.ca-run --config win.ini --steps 5 --strategy rl --model_dir model1CA will run using the config filewin.iniand  will aim to have a sequence of5 steps. It will also usereinforcement learningas strategy, with the pretrained modelmodel1.1b.ca-run --config web.ini --steps 15CA will run using the config fileweb.iniand  will aim to have a sequence of15 steps. In this case,
the strategy to get the next action is random.2. ca-compareIt is used to compare two test results of the same test sequence, applied to two versions of the same SUT.Parameters--sequence1_folder(optional) - recordings folder with theexpectedtest results--sequence2_folder(optional) - recordings folder with theactualtest resultsNote:If both sequences are missing, CA will attempt to find the last two test resultsResultsResults can be found withinsequence1_folderfolder in a dedicated sub-folder.Command examples2a.ca-compare --sequence1_folder 2022_08_26-12_14_34 --sequence2_folder 2022_08_26-12_24_45CA will compare the test results from defined sequence1_folder (expected) and sequence2_folder (actual).Configuration filesThis project uses.ini filesas configure files to change software's behavior.User must define own configuration files.Ini file must define following values:[driver]
  command_executor=
  platform=
  app=Example .ini file using WinAppDriver and Windows platform:[driver]
  command_executor=http://127.0.0.1:4723
  platform=win
  app=/path/to/test/application.exeExample .ini file using SeleniumDriver and Web platform:[driver]
  command_executor=http://127.0.0.1:4444/wd/hub
  platform=web
  url=https://learn.ivves.eu/Port of command_executor needs to match the driver's (ex. WinAppDriver/chromedriver) port.AcknowledgementThe package was developed byF-Secure Corporationin scope ofIVVES project. This work was labelled byITEA3and funded by local authorities under grant agreement 'ITEA-2019-18022-IVVES'.