Package: divinegift
The most useful package for you, young python programmer :)How to use itInstall this packages before usesqlalchemy            - For works with databasescx_Oracle             - Oracle driversqlalchemy-pytds      - MSSQL driverpsycopg2-binary       - Postgres driverxlutils               - For Excelxlsxwriter            - for Excelopenpyxl              - For Exceltransliterate         - For Transliterationconfluent-kafka[avro] - For work with kafkalogging# Import necessary functions
from divinegift.logger import log_info, log_err, log_warning, set_loglevel

# info msg
log_info('Your message')

# error msg
log_err('Error msg',
        src='Error source',  # e.g. str(sys.argv[0])
        mode=['telegram', 'email']  # May be empty
        channel={ 
            "telegram": -1001343660695, 
            "email": {"TO": ["your@domain.ru"], "FROM": "from@domain.com", 
                "HOST": "smtp.domain.com", "usr": "from@domain.com", "pwd": "supersecretpassword"}  
        })  # You can add "CC" to "email" for add copy_to addresses

# error msg with out sending problem to external system
log_err('Error msg', src='Error src')Pass log_level and log_name through cmd argumentsTo specify log_level and log_name in your app you can send it through arguments:if __name__ == '__main__':
    # Get all args from cmd:
    args = get_args()
    # Get log_level, log_name, log_dir
    lp = get_log_param(args)
    # Set log_level and log_name:
    set_loglevel(lp.get('log_level'), lp.get('log_name'), lp.get('log_dir'))You should pass your args by pairs: key value, e.g. --log_level INFO
Available variants for logging are:--log_level or shortcut is -ll--log_name or shortcut is -ln--log_dir or shortcut is -ldlog_level could be DEBUG, INFO, WARNING, ERRORExample of starting app with arguments:python test.py -ll INFO -ln test.logConfig parsingTo parsing configs you can use classdivinegift.config.Settings.
By default, it's use yaml as config language. But you can use json-style too.YAML-configfrom divinegift.config import Settings  # Necessary imports

settings = {}

# You should use divinegift.logger.set_loglevel before config parsing
s = Settings()
s.parse_settings('./settings.ini')
settings = s.get_settings()Config examplemonitoring:
- telegram
- email
- slack
monitoring_channel:
    email_to:
    - aims.control@s7.ru
    telegram: -1001343660695JSON-configfrom divinegift.config import Settings  # Necessary imports

settings = {}

# You should use divinegift.logger.set_loglevel before config parsing
s = Settings()
s.parse_settings('./settings.ini', use_yaml=False)
settings = s.get_settings()Config example{
    "log_level": "INFO",
    "log_name": "YourAwesomeProject.log",
    "monitoring": [
        "telegram",
        "email"
    ],
    "monitoring_channel": {
        "telegram": -1001343660695,
        "email_to": [
            "aims.control@s7.ru"
        ]
    }
 }Working with DB (sqlalchemy)You should define dict with db_conn creditional.
For example:OracleInstall oracle driver:pip install cx_oracledb_conn = {
    "db_user": "dbuser",             # username
    "db_pass": "dbpass",             # password
    "db_host": "dbhost",             # host (ip, fqdn). could be empty if we connect via tns
    "db_port": "",                   # port (string). could be empty if we connect via tns
    "db_name": "dbname",             # database name
    "db_schm": "",                   # db scheme if not equal username
    "dialect": "oracle"              # if use cx_Oracle or oracle+another_dialect
}MSSQLInstall mssql driver:pip install sqlalchemy-pytdsdb_conn = {
    "db_user": "dbuser",             # username
    "db_pass": "dbpass",             # password
    "db_host": "",                   # host (ip, fqdn). could be empty if we connect via tns
    "db_port": "",                   # port (string). could be empty if we connect via tns
    "db_name": "dbname",             # database name
    "db_schm": "",                   # db scheme if not equal username
    "dialect": "mssql+pytds"         # mssql dialect
}PostgresInstall postgres driver:pip install psycopg2db_conn = {
    "db_user": "dbuser",             # username
    "db_pass": "dbpass",             # password
    "db_host": "",                   # host (ip, fqdn). could be empty if we connect via tns
    "db_port": "",                   # port (string). could be empty if we connect via tns
    "db_name": "dbname",             # database name
    "db_schm": "",                   # db scheme if not equal username
    "dialect": "postgresql+psycopg2" # dialect
}Create connectionUse class DBClient to create connection to DB. Old-styled functions, which contained indivinegift.dbmodule directly, are deprecated but still works.from divinegift.db import DBClient
connection = DBClient(db_conn)            # db_conn - variable which was described above
# Describe which fields you wants to method get_conn will returned (possible fields are 'engine', 'conn' and 'metadata')
engine, conn, metadata = connection.get_conn(fields=['engine', 'conn', 'metadata'])If you need to call stored procedure with db cursors you should use raw connection.from divinegift.db import DBClient
connection = DBClient(db_conn, do_initialize=False)            # db_conn - variable which was described above
connection.set_raw_conn()
conn = connection.get_conn(fields='conn')Get data from sript (file or just string)After you got "connection" variable you can  get data from file or from str variable directly.from divinegift.db import DBClient
connection = DBClient(db_conn)

result = connection.get_data('path/to/scripts/some_script.sql')
# or you can use str variable:
script = 'select * from dual'
result = connection.get_data(script)
print(result)
>>>[{'dummy': 'X'}]You can use specific encoding for your files (by default it's 'cp1251').
Just put it into args:result = connection.get_data('path/to/scripts/some_script.sql', encoding='utf8')Also you can add some variables into your script (e.g. date) and then you can pass it into a function:from divinegift.db import DBClient
connection = DBClient(db_conn)

script = """select * from dual
where dummy = '$param'"""
parameters = {'param': 'X'}
result = connection.get_data(script, **parameters)
# Or another variant
result = connection.get_data(script, param='X')
print(result)
>>>[{'dummy': 'X'}]Run script without getting dataYou can run script without recieving data.
You should usedivinegift.db.DBClient.run_scriptfor this like get_data, e.g.:from divinegift.db import Connection
connection = Connection(db_conn)
connection.run_script('path/to/scripts/some_script.sql')Sending emailYou can use function send_mail from classdivinegift.sender.SenderYou should set your msg, subject and list of recipients, and account which should be used for sending email
Simple example:from divinegift.sender import Sender

sender = Sender()
sender.send_mail('Test message', 'Test subject', TO=['your@domain.com'],
           FROM="from@domain.com", HOST="smtp.domain.com", usr="from", pwd="pwd")You can specify TO, CC, BCC, HOST, FROM and attachments. Also you can send it like html-message or like text.You should pass list of attachments files with absolute path to it. You can use functiondivinegift.main.get_list_filesfor get it.
For sending email with attachment(s):from divinegift.main import get_list_files
from divinegift.sender import Sender

sender = Sender()
attachment_list = get_list_files('/path/to/files', add_path=True)
sender.send_mail('Hello! This are files in attach', 'Test sending attachments', ['your@domain.com'], 
                  FROM="from@domain.com", HOST="smtp.domain.com", usr="from", pwd="pwd",
                  attachments=attachment_list)
# Also you can send only one file:
attachment = '/path/to/file/file_name'
sender.send_mail('Hello! There is file in attach', 'File', ['your@domain.com'], 
                FROM="from@domain.com", HOST="smtp.domain.com", usr="from", pwd="pwd",
                attachments=attachment)If you set IS_HTML to False (by default it is True), you could send an email like simple text message, not htmlWork with JSONYou can simple parse and create JSONsTo create json you could usedivinegift.main.create_jsonTo parse it you could usedivinegift.main.parse_jsonOr you could use classdivinegift.main.Jsoninstead of itFor example:from divinegift.main import create_json, parse_json
A = {'key1': 'data1', 'key2': 'data2'}
create_json(A, 'json_file_name.json')
B = parse_json('json_file_name.json')

print(B)
>>> {'key1': 'data1', 'key2': 'data2'}

from divinegift.main import Json
A = {'key1': 'data1', 'key2': 'data2'}
json_obj = Json('json_file_class.json')
json_obj.set_data(A)
json_obj.create()
B = json_obj.parse()Work with YAMLYou can simple parse and create YAMLsTo create json you could usedivinegift.main.create_yamlTo parse it you could usedivinegift.main.parse_yamlOr you could use classdivinegift.main.Yamlinstead of itFor example:from divinegift.main import create_yaml, parse_yaml
A = {'key1': 'data1', 'key2': 'data2'}
create_yaml(A, 'yaml_file_name.yml')
B = parse_yaml('yaml_file_name.yml')

print(B)
>>> {'key1': 'data1', 'key2': 'data2'}

from divinegift.main import Yaml
A = {'key1': 'data1', 'key2': 'data2'}
yml_obj = Yaml('yml_file_class.yml')
yml_obj.set_data(A)
yml_obj.create()
B = yml_obj.parse()Transliterate strings between Russian and English and backFrom version 1.0.8 you can use transliterate library to transliterate strings between languagesExample:from divinegift.translit import translit

name = 'SHEVCHENKO ANDREY'
name_r = translit(name, 'ru_ext')
name_e = translit(name_r, 'ru_ext', reversed=True)
name_r_cap = translit(name, 'ru_ext').capitalize()
name_r_low = translit(name, 'ru_ext').lower()

print(f'From English to Russian: {name}\t->\t{name_r}')
print(f'From Russian to English: {name_r}\t->\t{name_e}')
print(f'Capitalize             : {name}\t->\t{name_r_cap}')
print(f'Lower                  : {name}\t->\t{name_r_low}')Code from above will show next:From English to Russian: SHEVCHENKO ANDREY  ->  ШЕВЧЕНКО АНДРЕЙ
From Russian to English: ШЕВЧЕНКО АНДРЕЙ    ->  SHEVCHENKO ANDREI
Capitalize             : SHEVCHENKO ANDREY  ->  Шевченко андрей
Lower                  : SHEVCHENKO ANDREY  ->  шевченко андрейEncryptionFrom version 1.0.10 you can use encryption moduleSimple exampleExample:from divinegift.cipher import get_key, get_cipher, encrypt_str, decrypt_str

cipher_key = get_key()
cipher = get_cipher(cipher_key)
text = 'qwerty1234!!'
text_enc = encrypt_str(text, cipher)
print(text_enc)
text_dec = decrypt_str(text_enc, cipher)
print(text_dec)Code above will output next:gAAAAABcanXfhUr9i__R_24rPyrHzZoMgQSTYiBmx9ZtVqdcMiGZPOxoSz4gkAW0Y9TDWpAJ6jzAjPo-mrK_IcJcdByyfWrbhQ==
qwerty1234!!If you use parameter get_str=False in functions encrypt_str and decrypt_str than this functions will returns binary stringWorks with key in fileSave your key in file bywrite_keyfunction:from divinegift.cipher import get_key, write_key

cipher_key = get_key()
write_key('key.ck', cipher_key)Read your key file byread_keyfunction:from divinegift.cipher import read_key

cipher_key = read_key('key.ck')CaesarYou can use caesar encrypt/decrypt:from divinegift.cipher import caesar_code

text = caesar_code('Hello, World!', shift=5)
print(text)It will output next:Mjqqt, 1twqi!Easy encription/decryption database-passwords for more securityBefore all you should encrypt your file with settings.
Use next code to do this once:from divinegift.config import Settings

s = Settings()
s.parse_settings('settings.conf')
s.initialize_cipher()
s.encrypt_password('db_conn')   # db_conn - name of db connection in settings.conf which contains "db_pass"
s.save_settings('settings.conf')After that your password in section 'db_conn' will automaticaly encrypted.
If you have more db-connections just add s.encrypt_password('db_conn_name_you_have') before saving functionNext you must use decryption function in your code to use connection:from divinegift.config import Settings

s = Settings()
s.parse_settings('settings.conf')
s.decrypt_password('db_conn')   # db_conn - name of db connection in settings.conf which contains "db_pass"Live templates. Start create your app as easy as possibleFrom version 1.0.11 you can create files from templates.
You should use moduletemplatorfor this.Example:
Create tmp.py with following text and run it:from divinegift.templator import Templator

t = Templator()
# create console app, or main logic (you can omit the file extension, '.py' will add automaticaly.):
t.create_console('your_awesome_name.py')
# or
t.create_console()  # it will create 'main.py' file

# create QT-app:
t.create_gui(your_awesome_name.py')
# or
t.create_gui()  # it will create 'main_gui.py' file

# create config file:
t.create_config('your_config_name.ini')
# or
t.create_config()       # it will create 'settings.ini' file

# After creating file with config you can add email section on it:
t.add_email_config('your_config_name.ini')
# or
t.add_email_config()    # it will add email section to 'settings.ini'Kafka ClientFrom version 1.3.0 you can use kafka client to read and write data from/to topicsExample:from divinegift.kafka_client import KafkaClient

kafka_client = KafkaClient()

# Reader
kafka_client.set_consumer(**s.settings.get('consumer_config'))

messages = kafka_client.read_messages(topic_name)
# You can read all messages from begin if you needed:
messages = kafka_client.read_messages(topic_name, from_beginning=True)

# Writer
kafka_client.set_consumer(**s.settings.get('producer_config'))

kafka_client.send_message(topic_name, msg)Config example from example above:kafka_config: &kafka_config
    bootstrap_servers:
    - server.domain:9093
    security_protocol: SSL
    ssl_check_hostname: False
    ssl_cafile: CARoot.pem
    ssl_certfile: certificate.pem
    ssl_keyfile: key.pem
producer_config:
    <<: *kafka_config
consumer_config:
    <<: *kafka_config
    consumer_timeout_ms: 1000Working with ExcelReading fileFor reading excel-file you should use functiondivinegift.excel.read_excelYou should pass filename, array with column namesOptional fields are:
sheet_name, int_columns, date_columns, start_rowfrom divinegift import excel

filename = 'your/path/to/excel.xlsx'      # or it could be xls
excel_header = ['column1', 'column2', ]
excel_data = excel.read_excel(filename, excel_header)By default, all cells are read as strings, but if you need read int columns/date columns, you could pass their names
in parameters int_columns/date_columns. You should name it like you pass it at excel_headerfrom divinegift import excel

filename = 'your/path/to/excel.xlsx'      # or it could be xls
excel_header = ['column1', 'column2', 'int_col', 'date_col']
excel_data = excel.read_excel(filename, excel_header, int_columns=['int_col'], date_col=['date_col'])Writing fileFor writing excel-file you should use functiondivinegift.excel.create_excelYou should pass filename and list with dataOptional fields are:
sheet_name, headerWhen you set excel_header, you could set column widthfrom divinegift import excel

filename = 'path/to/excel/your.xlsx'
data = [{'col1': 1, 'col2': 2,},]    # or it could be just list of list

excel_header = {'col1': 10, 'col2', 13, }
excel.create_excel(data, filename, excel_header=excel_header)