Package: easy-reports
Easy ReportsEasy Reports is a python package which simplifies report configuration, generation and distribution.
Reasons for this package creation are:removal of repetitive code from multiple python report scripts written by workers with different level of programming skillsability to extract data from two or more databasesability to join query results into one final datasetsimplification of excel file formatting by introducing dictionary bases configurationintroduction of @processor decorated functions that can do additional data processing and return dataset for excel generatorTable of ContentsEasy ReportsTable of ContentsInstallationUsageApplication ConfigurationReport ConfigurationReports LoadingReport GenerationExamplesCreation of Report boilerplateTestingContributingLicenseInstallationCreate project folder and initialize venv environment$mkdirmyproject
$cdmyproject
$python3-mvenvvenvWithin the activated environment, use follownig command to install Easy Reports$pipinstalleasy-reportsUsageA minimal Easy Report looks as below:fromeasy_reportsimportEasyReportfrompathlibimportPathapp=EasyReport(base_path=Path(__file__).resolve().parent)app.base_config.from_pyfile('settings.py')app.load_reports(report_list=['R001'])app.run()Application ConfigurationApplication default confinguration is defined in defaults.py.
Config parameters can be changed:defining environment variable prefixed with EASY_REPORTSpassed as keyword arguments to EasyReport()loaded from settings.py fileEach Report configuration is created on top of Application'sbase_config.
Selected parameters can be overriden by uppercase attribiutes in Report'sMeta.Report ConfigurationBy default each Report lives in dedicated folder inside path:app.defs_pathPreferred (default) Report structure consists of:report.py module file that contains Report definition such as settings, mappings andprocessorssubfolders:/sql- SQL scripts used to retrive data/templ- templates of email messages/cache- cached SQL script's results and dataframes used by Excel creator/logs- log files/arch- generated Excel report filesIt is not advised but if necessary default folder paths can be changed in Application configuration step.
For example change of default/archcan be done by:setting evironment variable$exportEASY_REPORTS_ARCH_BASE_PATH='/new_location/arch'passing keyword argument to EasyReport class constructorapp=EasyReport(...,arch_base_path='/new_location/arch')setting variable in settings.pyARCH_BASE_PATH='/new_location/arch'Reports LoadingReports loading is done by scaningapp.defs_pathfor subfolders containing report.py modules. Modules (Reports) are imported, configured and stored inapp._reportlistReport GenerationReports can be generated individually by calling:# generate selected Report 'R001'app.refresh(name='R001')or allapp.refresh_all()orapp.run()Additional keyword arguments can be passed to the above functions:
One of them, useful while preparing and developing new Report module isskiplist of blocked parts of Report generation flow# skip excel creation and email sendingapp.refresh(name='R001',skip=['excel','email'])ExamplesCreation of Report boilerplateNew Report boilerplate can be created using CLI command.$easy-reportscreate
Enternewreportsymbol:R7777
CreatingreportR7777...New subfolder R7777 will be created uderdefspath.
Report folder will contain report.py module file and four subfolders (as shown below)$cddefs
defs/$tree
.
└──R7777├──arch├──cache├──logs├──sql└──report.pyModule file report.py contains all configuration needed to generate report files.
Global (default) configuration from setting.py can be extended by each report.If needed additional aliases and connection strings can be defined as below:db_list={'alias_1':{'url':'testing'},'alias_2':{'url':'testing'},# 'alias_3': {'url': 'testing'}}List of SQL files with DB aliases and list of fields used to join resulting Dataframes into one aggregate result.sql_list=[('sql_1','alias_1',['PK']),('sql_2','alias_2',['PK']),# ('sql_3', 'alias_3', ['PK']),]Reports can be sent by email. Email configuration is done in the dictonary shown below. Each email is bound with report files by list of ID'semail_config={1:{'to':'recipient@email.com','cc':'','subject':'Test','onbehalf':'','body':'This is test','template':'','attachments':[],'attachments_rpt_id':[1],}}Reports configuration is done in the dictionary shown below. Each entry is ID of the report.
Report config defines such elements as report filneme, wheather is can be sent by email (added to attachments dictionary), list of sheets. Each sheet must be named and bound to data source (name of Dataframe). Other dictionary keys allow configuration and formatting of the resultig sheet.rpt_config={1:{'filename':'report_{symbol}_{date}.xlsx','send_email':True,'sheets':{1:{'sheet_name':'Result','data_src':'df_result','output_columns':[],'freeze_panes':'B2','header_formats':{'*':'header',},'column_formats':{'column_name':'fg_excel_35',},'column_options':{'D:F':{'level':1,'hidden':True},'C:C':{'collapsed':True},},'conditional_formats':{'column_name':[{'type':'data_bar','bar_color':'#50DAF6','bar_border_color':'black',},],},'data_validations':{'column_name':{'validate':'list','source':['Excelent','Good','Moderate','Poor'],},},}},}}Additional processing can be done to aggregate result Dataframe (df_final) or any other stored in self.results dictionary. Functions decorated with @processor are executed in order of definition in the report.py module file. Each function should return dictionary of Name and Dataframe object witch will be added to self.results dictionary.@processordefp01(self)->dict:df=self.results['df_final']# dataframe processingreturn{'df_result':df}Default settings can be overriden by defining variables in the Meta class body:Default subfolder names can be modified by defining any of below variablesclassReport(ReportBase):classMeta:LOGS_DIR='logs'ARCH_DIR='arch'SQL_DIR='sql'CACHE_DIR='cache'TEMPL_DIR='templ'...If necessary full paths can be provided that will override any other path and folder settingsclassReport(ReportBase):classMeta:logs_path=pathlib.Path('/custom_path/logs').resolve()arch_path=pathlib.Path('/custom_path/arch').resolve()sql_path=pathlib.Path('/custom_path/sql').resolve()cache_path=pathlib.Path('/custom_path/cache').resolve()templ_path=pathlib.Path('/custom_path/templ').resolve()...TestingPackege can has been tested against three databases: Postgres, MySQL and SqlServer.
Test can be run by deploying stack of Docker containers.Running command bellow will start container in detached mode and run pytest in the app contaner when all db container are ready.$dockercomposeup-dWhile contaner are running in the background tests from host machine can be run. Prior to running pytest or tox aliases to localhost for db-postgres, db-mysql, db-mssql must be added to enable host name resolution.ContributingTo contribute to the project, you can:Submit bug reports and feature requests through the GitHub issue trackerFork the repository and submit pull requests with bug fixes or new featuresHelp improve the documentationPlease read the contributing guidelines for more information.LicenseThis project is licensed under the MIT License. See the LICENSE file for details.