Package: prosto
____                _        
|  _ \ _ __ ___  ___| |_ ___   _________________________________________________
| |_) | '__/ _ \/ __| __/ _ \ 
|  __/| | | (_) \__ \ || (_) | Functions matter! No map-reduce. No join-groupby.
|_|   |_|  \___/|___/\__\___/  _________________________________________________вЂўDOCUMENTATIONвЂўWhy Prosto?вЂўQuick startвЂўHow to useвЂўReferencesвЂўProstois a Python data processing toolkit to (programmatically or usingColumn-SQL) author and execute complex data processing workflows. Conceptually, it is an alternative to purelyset-orientedapproaches to data processing like map-reduce, relational algebra, SQL or data-frame-based tools likepandas.Prostoradically changes the way data is processed by relying on a novel data processing paradigm:concept-oriented modelof data[2]. It treats columns (modelled via mathematical functions) as first-class elements of the data processing pipeline having the same rights as tables. If a traditional data processing graph consists of only set operations than theProstoworkflow consists of two types of operations:Table operationsproduce (populate) new tables from existing tables. A table is an implementation of a mathematicalsetwhich is a collection of tuples.Column operationsproduce (evaluate) new columns from existing columns. A column is an implementation of a mathematicalfunctionwhich maps tuples from one set to another set.An example of such aProstoworkflow consisting of 3 column operations is shown below. The main difference from traditional approaches is that thisProstoworkflow will not modify any table - it changes only columns. Formally, if traditional approaches apply set operations by generating new sets from already inferred sets, thenProstoderives newfunctionsfrom existing functions. In many cases, using functions (column operations) is much simpler and more natural.Prostoprovides two ways to define its operations:Programmaticallyby calling function with parameters specifying an operationColumn-SQLby means of syntactic statements with all operation parameters. Column-SQL is a new way to define a column-oriented data processing workflow. It is a syntactic alternative to programmatic operations. Read more here:Column-SQLProstooperations are demonstrated in notebooks which can be found in the "notebooks" folder in the main repo. Do your own experiments by tweaking them and playing around with the code:https://github.com/asavinov/prosto/tree/master/notebooksThe column-oriented approach was used in the Intelligent Trading Bot for deriving new features:https://github.com/asavinov/intelligent-trading-botMore detailed information can be found in the documentation:http://prosto.readthedocs.ioMotivation: Why Prosto?Why functions and column-orientation?In traditional approaches to data processing, we frequently need to produce a new table even though we need to define a new attribute. For example, in SQL, a new relation has to be produced even if we want to define a new calculated attribute. We also need to produce a new relation (using join) if we want to add a column with data from another table. Data aggregation by means of groupby operation produces a new relation too, although the goal is to compute a new attribute.In many important cases, processing data usingonlyset operations is counter-intuitive, and this is why map-reduce, join-groupby (including SQL) and similar set-oriented approaches require high expertise and are error-proneThe main unique novel feature ofProstois that it relies on a different formal basis:Prostoadds mathematicalfunctions(implemented as columns) to its model by significantly simplifying data processing and analysisNow, if we want to define a new attribute then we can do it directly without defining new unnecessary tables, collections or relations. New columns are defined in terms of other columns (in different tables) and this makes this model similar to how spreadsheets work but instead of cells we use columns. For comparison, if in spreadsheets we could define a new cell asA1=B2+C3, then inProstowe could define a new column asColumn1=Column2+Column3. The main theoretical challange is in introducing a set of column operations between columns inmultipletables in such a way that these operations effectively replace relational operations (join and groupby) and cover most important use cases. How it is done is described in[2].ProstowithColumn-SQLis one possible implementation of this model.More info:Why functions and column-orientation?Unique features and benefits of ProstoProstoprovides the following unique features and benefits:Easily processing data inmultiple tables. New derived columns are added directly to tables  without creating multiple intermediate tablesGetting rid of join and group-by. Column definitions such aslink columnsandaggregate columnsare used instead of join and groupby set operationsFlexibility and modularization via Pythonuser-defined functions(UDF). UDFs describe what needs to be done with the data only in this operation using arbitrary Python code. If UDF for an operation changes then it is not necessary to update other operationsParameterization of operations by amodel object. A model can be as simple as one value and as complex as a trained deep neural network. This feature leads to a novel view of how data analysis should be organized by combining feature engineering and machine learning so that both model training and model use (predictions or transformations) are part of one data processing workflow. Currently models are supported only as static parameters but in future there will be a possibility to train model within the same workflowFuture directions. Incremental evaluation and data dictionaryMore info:Benefits of ProstoQuick start (using Column-SQL)Creating a workflowAll data elements (tables and columns) as well as operations for data generation are defined in a workflow object (interpreted as a context):importprostoasprprosto=pr.Prosto("My Prosto Workflow")More info:Workflow and operationsPopulating a source tableEach table has some structure which is defined by itsattributes. Table data is defined by the tuples it consists of and each tuple is a combination of some attribute values.The simplest way to populate a source table is to create or load apandasdata frame and then pass it to a Column-SQL statement:sales_data={"product_name":["beer","chips","chips","beer","chips"],"quantity":[1,2,3,2,1],"price":[10.0,5.0,6.0,15.0,4.0]}sales_df=pd.DataFrame(sales_data)prosto.column_sql("TABLE Sales",sales_df)The Column-SQL statementTABLE Saleswill create a definition of a source table with data from thesales_dfdata frame.In more complex cases, we could pass auser-defined function(UDF) instead of the data frame. This function is supposed to "know" where to load data from by returning apandasdata. For example, it could load data from acsvfile.More info:Table operationsDefining a calculate columnA column is formally interpreted as a mathematical function which maps tuples (defined by table attributes) of this table to output values. The simplest column operation is acalculatecolumn whichcomputesoutput values using the values of the specified input columns of the same table:prosto.column_sql("CALCULATE  Sales(quantity, price) -> amount",lambdax:x["quantity"]*x["price"])This newamountcolumn will store the amount computed for each record as a product ofquantityandprice. TheCALCULATEstatement consists of two parts separated by arrow:First, we define the source table and its columns that we want to process as input:Sales(quantity, price)Second, we define a column to be created:amountThis use of arrows is an important syntactic convention of Column-SQL which informally represent a flow of data within one table or between tables.More info:Column operationsExecuting a workflowA workflow object stores only operationdefinitions. In order to really process data, the workflow has to be executed:prosto.run()Prostotranslates a workflow into a graph of operations (topology) taking into account their dependencies and then executes each operation.Now we can explore the result by reading data form the table along with the calculate column:df=prosto.get_table("Sales").get_df()print(df)product_name quantity price amount
0  beer         1        10.0  10.0
1  chips        2        5.0   10.0
2  chips        3        6.0   18.0
3  beer         2        15.0  30.0
4  chips        1        4.0   4.0Theamountcolumn was derived from the data in other columns. If we change input data, then we can again run this workflow and the derived column will contain updated results.The full power ofProstocomes from the ability to process data inmultipletables by definining derived links (instead of joins) and then aggregating data based on these links (without groupby). Note that both linking and aggregation do not require and will not produce new tables: only columns are defined and evaluated. For example, we might use column paths likemy_derived_link::my_columnin operations in order to access data in other tables.More info:Column-SQLHow to useInstall from source codeCheck out the source code and execute this command in the project directory (wheresetup.pyis located):$pipinstall.Or alternatively:$pythonsetup.pyinstallInstall from PYPIThis command will install the latest release ofProstofrom PYPI:$pipinstallprostoHow to testRun tests from the project root:$python-mpytestor$pythonsetup.pytestReferences[1]: A.Savinov. On the importance of functions in data modeling, Eprint:arXiv:2012.15570[cs.DB], 2019.https://www.researchgate.net/publication/348079767_On_the_importance_of_functions_in_data_modeling[2]: A.Savinov. Concept-oriented model: Modeling and processing data using functions, Eprint:arXiv:1606.02237[cs.DB], 2019.https://www.researchgate.net/publication/337336089_Concept-oriented_model_Modeling_and_processing_data_using_functions[3]: A.Savinov. From Group-By to Accumulation: Data Aggregation Revisited, Proc. IoTBDS 2017, 370-379.https://www.researchgate.net/publication/316551218_From_Group-by_to_Accumulation_Data_Aggregation_Revisited[4]: A.Savinov. Concept-oriented model: the Functional View, Eprint:arXiv:1606.02237[cs.DB], 2016.https://www.researchgate.net/publication/303840097_Concept-Oriented_Model_the_Functional_View[5]: A.Savinov. Joins vs. Links or Relational Join Considered Harmful, Proc. IoTBD 2016, 362-368.https://www.researchgate.net/publication/301764816_Joins_vs_Links_or_Relational_Join_Considered_Harmful