Package: yarm
yarmYarm: Yet Another Report Maker.FeaturesYarm makes it easy for you to createrecurring reportsby:Importingmultiple spreadsheets and CSVsinto a temporary database.Offeringeasyoptions forcommon data cleaningtasks (e.g.replace,slugify_columns,pivot)RunningSQL queries(or, forpandasfans,customPythoncode) on all this data.Exporting the resultsas a newspreadsheet,CSV, or even SQLitedatabase.All configured in asimpleYAML filefor easyreuse. Download fresh data,yarm run, and you're done.Basic UsageFirst Time You Run a New ReportCollect your XLSX and/or CSV data files into a directory for this report.Initialize a new YAML config file:$yarmnewEdit the YAML config file (see below).Configure your input spreadsheets and CSV files as tables.Write one or moreSELECTqueries on these tables to create output sheets.(Optional) Need advanced manipulation of your data? Writepandascode in a separate.pyfile.Run the report:$ yarm runSend the output spreadsheet to your boss/client/head of state. Was it really that easy?Every Subsequent TimeCollect fresh data. Save it over the old files.Run the report.$ yarm run -fSend the output spreadsheet.Take the afternoon off.Advanced UsagePlease see the extensivedocumentationfor more details and features.Example Report Config FileYou configure a report in asingle YAML file.Each query becomes a separate sheet in your output spreadsheet.This example config file is moderately complex. Your report can be much simpler; you might have only one or two tables and a single query. (Or you might have ten queries, each with acustom postprocess function...)---output:dir:Outputbasename:Sales_Report# Optional input options (more are available):input:slugify_columns:truelowercase_columns:true# Set up your data sources:tables_config:# CSV file: the easiest data source.products:-path:Products.csv# Spreadsheet: You need both the path and the sheet name.orders:-path:Orders.xlsxsheet:Orders# You can import different sheets as separate tables.order_details:-path:Orders.xlsxsheet:Order Details# You can combine multiple data sources into a single table,# as long as their columns can be merged.tax:-path:Sales Tax Rates Northeast.xlsxsheet:NY-path:Sales Tax Rates Northeast.xlsxsheet:PA-path:TAXES_SOUTH.csv# Set up your output spreadsheet:queries:-name:Order Details with Product Namessql:SELECT * FROM order_details as od JOIN products as p ON od.product_id = p.id;-name:Orders With Sales Taxsql:>SELECT orders.*,tax.rateFROM ordersJOIN taxON orders.billing_state = tax.state;# These query results will need a Python function to complete this sheet:postprocess:calculate_tax# But first, we can do simple regex replacements right here:replace:billing_state:Virginia:VAWest Virginia:WV# Since we need that custom function calculate_tax(), we'll# write it in a separate Python file.import:-path:custom.pyRead more aboutbasic configurationandadvanced options.Custom Postprocessing CodeIf the power of SQL and make-it-easy options likeslugify_columnsaren't enough for you, you can write acustom postprocess functionfor any query you like.Status: Alpha (Try It!)Yarm is currently inalpha. Core features areworkingand thoroughlydocumented.I rely onyarmfor my own recurring reports.If you are desperate to stop doing a recurring report by hand, giveyarma try.If something breaks, or if you have any suggestions or comments, pleasefile an issue. I'd love to hear what you think.For upcoming features, see theRoadmap.RequirementsPython 3.7 or laterA terminalOne or more spreadsheets that you want to querySomething to do with all this impending free time...InstallationYou can installyarmviapipfromPyPI:$pipinstallyarmBut sinceyarmis a command line tool, you may prefer the excellentpipx:$pipxinstallyarmDocumentationComplete,extensivedocumentation is atyarm.readthedocs.io.Dive right in.Isyarmfor You?This tool has a clear focus: Make iteasyto run andrerun reportsfrom thecommand linethat querymultiple sourcesof tabular data.Once you set up the initial configuration file, the workflow for future reports is simple. Download fresh data over the old files, then rerun the report.This means thatyarmisnota tool for dataexploration.True, you may still wantyarmtoprepareyour data for exploration. Once you get used to listing a few data sources, setting a few options, and spitting out a nice, clean SQLite database or set of CSV files to play with, you may get hooked.But for iterative tinkering with your data, you're going to need other tools.Other Open Source Tools You Might Prefersqlitebrowser: An excellent GUI for exploring your SQLite database. I sometimes use this tofigure out my queriesbefore I save them into my config file.Jupyter Lab: If you find your SQL queries getting more and more arcane and complex, it's probably time to learnpandas, and that means unleashing the power of thisinteractive "notebook". Some reports are so complex that they really deserve to be run step-by-step, with immediate output after every command. Jupyter Lab makes that absurdly easy... and repeatable.SQL Notebook: A newer offering that I haven't used yet, but it looks like an interesting GUI combination of sqlitebrowser and a "Jupyter-style notebook interface". Could be very powerful.For quick,one-offdata manipulations on thecommand line, you can reach for excellent tools likejqfor JSON,mlrfor CSV, and evenhtmlqfor HTML. But once the command gets long and complex enough that you want to save it to a script, you might start missing SQL queries andyarmfeatures likeslugify_columns: true.ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theApache 2.0 license,Yarmis free and open source software.IssuesIf you encounter any problems, pleasefile an issuealong with a detailed description.CreditsThis project was generated from@cjolowicz'sHypermodern Python Cookiecuttertemplate.