Package: kururu
kururu - data science in the classroomWARNING: This project will undergo major changes in the next rewrite.InstallationExamplesEvaluated trainingfromaiunaimport*fromkururuimport*d=dataset("abalone").data# Each imported step is a callable object which can be used with no parameters.steps=binarize*split*pca*svm*metric# After the Data object goes through the steps, its last version has the test accuracy value at 'r'.d2=d>>stepsprint(d2.r)"""[0.56698565]"""Essential conceptsThe kururu framework simplifies data-related tasks (like data science)
by providing straight-forward tools for carefully chosen general concepts:step- kururu unifies all data-related processes under the concept of (data science)stepsteps are instances of Python classes derived from Stepsteps can be partially configured (e.g.,svm(kernel="poly"))or not configured at all (e.g.,svm()orsvmfor short),which has two different meanings, depending on the use case:a step ready to process data, using default values for the omitted parameters, e.g.:result=data>>svm(kernel="poly")# more on `>>` lattera sampleable (ordered1) set of different steps, e.g.:svms=svm(kernel="poly")# take the subset of all polynomial kernel SVMssvm0=~svms# sample a single configuration randomly, more on `~` latterresult=data>>svm0data- all input to (and output from) each step is a Data objectwhen in a machine learning context, this includes both training and test sets, and the results as wellinner/outer- steps that process two data sets expect the (outer) Data object to contain aninnerdata fieldstream- steps that process several data sets (e.g., partitioned or streamed) at once expect theouterdata to
contain astreamfieldoperator- steps are combined by operators and applied to Data objects through operatorsproduct- steps are chained by the*operator, which, analogously to the previousstepdefinition,
has two different meanings, depending on the use case:a sequence of steps (i.e., aProductobject) ready to process data, using default values for the omitted
parameters, e.g.:sequence=pca*svm(kernel="poly")result=data>>sequence# more on `>>` lattera sampleable set of different sequences of steps, e.g.:sequences=pca*svm(kernel="poly")# take the subset of all combinations between all PCAs and polynomial kernel SVMssequence=~sequences# sample a sequence randomly, more on `~` latterresult=data>>sequenceONGOING WORK FROM HERE UNTIL THE END OF THE PAGE....union- sets of steps are united by the+operator, which returns aUnionobject.
The implementation of the class Union doesn't follow the math concept of union of sets strictly.
It can have repeated elements for three reasons:
there is no gain in enforcing such math requirement;
it can be useful in some scenario involving thestreamfield where the repetition is actually needed;
and, more importantly, it is ordered (this applies to all sets if steps also).
Like any Step object, Union has two different use cases:a set of steps ready to process data, using default values for the omitted parameters, e.g.:ONGOINGWORK....union=mlp+svm(kernel="poly")result=data>>...*union*# more on `...` and `>>` lattera sampleable set of different sequences of steps, e.g.:sequences=pca*svm(kernel="poly")# take the subset of all combinations between all PCAs and polynomial kernel SVMssequence=~sequences# sample a sequence randomly, more on `~` latterresult=data>>sequence1Not important for the context, but it is worth to mention it is an ordered set,
because the mathematical definition of sets has no ordering.
We use the term "set" instead of list to avoid the need to explicitly borrow all the needed
concepts related to set operations.Data exploration versus machine learningWhether in a console (e.g.,ipython) or a temporary script (e.g.,jupyter notebooks), sometimes one needs to easily manipulate data,
or to easily set up a machine learning workflow.
To fit the needs of both distinct use cases (lightly leaning towards the writing of workflows), kururu provides step suffixes.
Some representative examples are given in the following table whereXXX_means the step XXX will be trained and tested on outer2data (useful for data exploration);in all other cases, XXX will be trained on inner data:XXXimeans XXX tested oninnerdataXXXomeans XXX tested onouterdataXXXbmeans XXX tested on bothinnerandouterdataUse caseNameShorthand(if any)Suffixed FormTraining setTest set(s)data explorationPrincipal Component AnalisysPCA_outer2outer2machine learningNoise ReductionNRNRiinnerinnermachine learningSupport Vector MachinesSVMSVMoinneroutermachine learningPrincipal Component AnalisysPCAPCAbinnerboth2Please note that, to simplify the terminology across the documentation,
the Data object is called "outer" even if it has no inner field.Finally, for completeness, the other meaningful versions of the steps presented in the previous table are now also included as follows.Use caseNameClass to useTraining setTest set(s)data explorationPrincipal Component AnalisysPCA_outeroutermachine learningPrincipal Component AnalisysPCAinnerbothdata explorationNoise ReductionNR_outeroutermachine learningNoise Reduction3NRinnerinnermachine learningSupport Vector MachinesSVMinneroutermachine learningSupport Vector MachinesSVMbinnerboth3Some of the steps (e.g., NR) may not implement the XXXb (e.g., NRb) version
when it is not applicable.Implementing custom Step classesAll fields that update a Data object should be lazy, i.e. should be a 'callable' without args.
An easy way to do that is to putlambda:before the value intended to be returned.
However, usually the return value is dependent upon fields from a previous Data object.
They should be accessed only from within the callable/lambda for a proper (iterator-safe) implementation.
Example:Creating a custom stepfromakangatu.distepimportDIStep# DIStep means "Data Independent Step", i.e. it does not depend on previously known data.classMyAdditionStep(DIStep):"""Multiplies the given field by a factor."""def__init__(self,field,factor):# All relevant step parameters should be passed to super() as keyword arguments.super().__init__(field=field,factor=factor)# Instance attributes are set as usual.self.field=fieldself.factor=factordef_process_(self,data):# All calculations (including access to data fields)#   is deferred to a future access to the return field - R in this case.returndata.update(self,R=lambda:data[self.field]*self.factor)Field name rulesA field named with a single letter has a lower case shortcut for automatic conversion from matrix to scalar/vector and vice versa.
Suffixes: ...
Reserved names: ...ContributionNothing clear yet, but one of the ways one can contribute is by creating their own repository (to be listed here as a
partner), using this repository (and/or other related ones) as a dependence. Monkey-patch can be used if one needs to
urgently integrate a module inside the same class tree used here, or ask for access to this repository, or submit a pull
request. The software architecture was planned taking that into account. It provides clear interface-classes to guide
the implementer/IDE, and each repository with a specific well-defined purpose.GrantsPart of the effort spent in the present code was kindly supported by Fapesp under supervision of
Prof. André C. P. L. F. de Carvalho at CEPID-CeMEAI (Grants 2013/07375-0 – 2019/01735-0).HistoryExcept dependencies like sklearn and other libraries, the novel ideias presented here are a result of a years-long
process of drafts, thinking, trial/error and rewrittings from scratch in several languages from Delphi, passing through
Haskell, Java and Scala to Python. The fundamental concepts were lightly borrowed from basic category theory concepts
like algebraic data structures that permeate many recent tendencies, e.g., in programming language design.For code (and academic) details refer to the following projects (few of them are usable by now):2003  [TUPI imaging: there was no widespread use of git at that time / retroactive repo yet to be created]2006  [Multicore NN: there was no widespread use of git at that time / retroactive repo yet to be created]2013 Functional language parser/interpreterhttps://github.com/davips/lamdheal-j2014 Machine learning library including Weka algorithms, optimized immutable data structure and models, hand-made
BLAS/LAPACK neural networks, transparent distributed processing (in conjunction with active-learning-scala), plotting,
evaluation, early replicabilityhttps://github.com/davips/mls2015 Active learning libraryhttps://github.com/davips/active-learning-scala2016 Thesis and dataset generation and
visualizationhttps://github.com/davips/tesehttps://github.com/davips/knowledge-boundaryhttps://github.com/davips/image2arff2018 Gaussian processseshttps://github.com/davips/surface2019 Client to generate reports from stored resultshttps://github.com/davips/mysql2csv2020 Python project where previous attempts and evolving ideias were
testedhttps://github.com/davips/pjml-may_archived