Package: nanbi
NanbiNanbiquara: speech of smart people, of clever peopleTranslated from theTupi Guarani Illustrated DictionaryNanbi is a framework that allows you to define data transformations in a composable way, agnostic of data processing engine (Pandas,  mySQL, Spark etc).Its syntax issql-like, inspired by PySpark and Scala-Spark approachesIt allows you to define a set of data transformations in a more composable way than SQL, for example, allowing for better readability specially on complex queriesIt allows you to execute your data transformations definitions in multiple engines (Pandas, mySQL, Spark etc) without having to change the data transformation definitionNanbi is right now under the initial stages of development. It's not fully ready for a version 1. So far, there is no compatibility with engines other than Pandas.Please get in touch if you have interest in using Nanbi on your work or personal project. Feature requests are welcome.SetupWhile the library isn't published in PyPIClone the repoCreate a symlink to the repoTODO(eemery): Add installation details once package gets published in PyPIGetting Started1. Creating a DataFrameNanbi uses the concept of aDataFrameto represent a table and its annotations (or metadata). Currently, Nanbi supports the creation of DataFrames from Pandas DataFrames and CSV files (using Pandas behind the scenes).From a Pandas DataFrameimportpandasaspdimportnanbi.connectors.pandasasnbpandas_df=pd.DataFrame({"num_a":[10,50,20,50,20],"num_b":[41,51,21,31,11]})df=nb.from_data_frame(pandas_df)From a CSV file (with Pandas)importnanbi.connectors.pandasasnbdf=nb.from_csv("path/to/my-file.csv")Viewing your imported dataTo visualize the imported or created data, just use the.display()method:importnanbi.connectors.pandasasnbdf=nb.from_csv("path/to/my-file.csv")df.display()The output will be a Pandas DataFrame, for example:col_a col_b
0 50    51
1 50    31
2 20    21
3 20    11
4 10    512. Enriching tables (.with_columns())Nanbi goal is to allow you to define data transformations to enrich your table with derived data in a composable way. One of the main ways that you can achieve this, is by the use of the.with_column()method. It creates a new column in your table according to the transformation formula you gave it. For example:importnanbi.connectors.pandasasnbdf=nb.from_csv("path/to/my-file.csv")enriched_df=df.with_column("result",col("col_a")+col("col_a"))enriched_df.display()The output will be a Pandas DataFrame in the form of:col_a col_b result
0 50    51    101
1 50    31    81
2 20    21    41
3 20    11    31
4 10    51    61Chaining TransformationsOne improvement that we can make to the code above is to take advantage of chaining transformations. We could have written the above code like:importnanbi.connectors.pandasasnbdf=nb.from_csv("path/to/my-file.csv").with_column("result",col("col_a")+col("col_a"))df.display()Improving Transformations Readability and ReusabilityAnother improvement that can be done, specially when transformations get complex, is to move the formula definition (i.e.,col("col_a") + col("col_a")) to its own variable. In the code above, this would look like:importnanbi.connectors.pandasasnbmy_complex_formula=col("col_a")+col("col_a")df=nb.from_csv("path/to/my-file.csv").with_column("result",my_complex_formula)df.display()