Package: db-extractor
db-extractorCode quality analysis and Build StatusWhat is this repository for?Extract information from databases (MySQL, MariaDB, SAP HANA to start with, other will be implemented later) using a combination of:extraction sequences file (JSON format) that is easy enough to create and maintain but also provide very complex features to be set;source system file (JSON format) to keep a central list of servers and/or databases to connect to that can be shared between people;user settings file (JSON format) to keep a central list of credentials that is not to be shared with anyone or maybe with a small group of people;Features implementedAbility to extract from a single source system or multiple using 1 JSON extraction sequence file;Ability to extract a single or multiple query for each source system using same JSON extraction sequence file;Ability to extract a single or multiple files using sessions for each query where parameters can be specified (currently on CSV and Excel file format are supported, other will follow);Multi-language (English, Italian, Romanian);Enhance behaviour choices so that besides existing 'skip-if-output-file-exists' and 'overwrite-if-output-file-exists' to have the option to specify to overwrite but only if the file is older than any choice of a CalculatedDate expression is given, as this is very useful when extracting large amount of data over VPN in small pieces and VPN drops (could mean already extracted pieces would be already skipped as not older than threshold imposed);Supported File Types/FormatsComma Separated Values (with ability to specify a custom separator of your preference)Excel 2013+JSONParquet (with compression algorithms: brotli, gzip, snappy and none to choose from)Pickle (with compression algorithms: bz2, gzip, xz, zip and none to choose from with as special value as "infer" to detect automatically the correct one from provided file extension)Who do I talk to?Repository owner is:Daniel PopiniucInstallationInstallation can be completed in few steps as follows:Ensure you have git available to your system:$ git --versionIf you get an error depending on your system you need to install it.For Windows you can do so fromGit for Windows;Download this project from Github:$ git clone https://github.com/danielgp/db-extractor <local_path_of_this_package>conventions used:<content_within_html_tags> = variables to be replaced with user values relevant stringsCreate a Python Virtual Environment using following command executed from project root folder:$ python(.exe) -m venv <local_folder_on_your_computer_for_this_package>/virtual_environment/Upgrade pip (PIP is a package manager for Python packages) and SetupTools using following command executed from newly created virtual environment and Scripts sub-folder:$ <local_path_of_this_package>/virtual_environment/Scripts/python(.exe) -m pip install --upgrade pip
    $ <local_path_of_this_package>/virtual_environment/Scripts/pip(.exe) install --upgrade setuptoolsInstall project prerequisites using following command executed from project root folder:$ <local_path_of_this_package>/virtual_environment/Scripts/python(.exe) <local_path_of_this_package>/setup.py installEnsure all localization source files are compile properly in order for the package to work properly$ <local_path_of_this_package>/virtual_environment/Scripts/python(.exe) <local_path_of_this_package>/sources/localizations_compile.pyMaintaining local package up-to-dateOnce the package is installed is quite important to keep up with latest releases as such are addressing important code improvements and potential security issues, and this can be achieved by following command:$ git --work-tree=<local_path_of_this_package> --git-dir=<local_path_of_this_package>/.git/ --no-pager pull origin masterconventions used:<content_within_html_tags> = variables to be replaced with user values relevant stringsUsage$ python <local_path_of_this_package>/sources/extractor.py --input-source-system-file <input_source_system_file_name> --input-credentials-file <input_credentials_file_name> --input-extracting-sequence-file <input_extracting_sequence_file_name> (--output-log-file <full_path_and_file_name_to_log_running_details>)conventions used:(content_within_round_parenthesis) = optional
<content_within_html_tags> = variables to be replaced with user values relevant strings
single vertical pipeline = separator for alternative optionsExample of usage$ python sources/extractor.py --input-source-system-file samples/sample---server-config.json --input-credentials-file samples/sample---user-settings.json --input-extracting-sequence-file samples/sample---list-of-fields.json --output-log-file samples/sample---list-of-fields.logCode of conductUseCODE_OF_CONDUCT.mdFeatures already raisedImplement ability to store extracted result-set into HTML format file;Features to request templateUsefeature_request.mdBug report templateUsebug_report.mdRequired software/drivers/configurationsseereadme_software.mdUsed referencesseereadme_reference.md