Package: ds-box
DS BoxPackage made byV. Levorato.This package provides:specificOperators for Apache Airflowin order to manage Data Science workflowspropose abunch of algorithms and routinescomplementary to Data Science usual packagesInstallDSbox can be installed from PyPI or by cloning this repo, and adding it to your project path.pip install ds-boxAirflow Operators and Data UnitsTwo specifics Airflow operators are given here:DataOperatorandKubeDataOperatorAirflow DataOperatorThis operator is able to add input and/or output data during the operator execution. As Airflow operator are stateless, passing simple datasets to be read or written through, for instance, a PythonOperator, needs to write all the code inside the operator to achieve that. With a DataOperator, the only contract you have is to manipulate Pandas dataframe(s) inside the function passed to it. Here is the code structure:fromdsbox.operators.data_operatorimportDataOperatormy_dag_task=DataOperator(operation_function=my_function,input_unit=reading_some_csv_unit,output_unit=writing_some_parquet_unit,dag=dag,task_id='My_process')With this example, I just need to definemy_function, theinput_unitand theoutput_unit.fromdsbox.operators.data_unitimportDataInputFileUnit,DataOutputFileUnitdefmy_function(dataframe):# useless function sorting dataframe by indexreturndataframe.sort_index()reading_some_csv_unit=DataInputFileUnit('path/to/dataset.csv',pandas_read_function_name='read_csv',sep=';')writing_some_parquet_unit=DataOutputFileUnit('path/to/index_sorted_dataset.parquet',pandas_write_function_name='to_parquet')Several pre-builtDataUnitare avalaible and you can easily build yours. Some examples gives you a preview of complete dags workflow for data science.DataGlobalInputUnitandDataGlobalOutputUnitcan now be used withany dataframe backend API (Pandas, Dask, Vaex, ...)as you define explicitly which function you are using.Airflow KubeDataOperator (KDO)UsingDataOperatorcan lead to some architecture issues: tasks must run on a machine or VM having the airflow worker service, and even if it should be moved to specific instance(s), it can be tricky to set auto-scaling mechanism, and to well manage your ressources.To avoid that, you can use containers to execute your tasks in a Kubernetes cluster, with the same definition asDataOperator, usingKubeDataOperator. This operator is based on theKubernetesPodOperator, and thought for data scientists.The difference is that the operation definition is made in a YAML description file, and all you have to do is call the definition with this operator. It takes also all the Kubernetes configuration in one parameter (could be done with**kwargsbut explicit is better). Same example as above with KDO:fromdsbox.operators.kube_data_operatorimportKubeDataOperatormy_dag_task=KubeDataOperator(operation='My_process',kube_conf=build_kpo_config(),dag=dag)Thebuild_kpo_config()function should be defined based on the project specificity and should explicitly set thecmdsandargumentsparameters needed by the underlyingKubernetesPodOperator, in addition to the cluster configuration parameters. Thecmdsshould call a function which will execute the task by using a class calledDataoperationscontaining all the datasets definitions with data units and operation functions. Example of a function called bycmds:fromdsbox.kube.data_operationsimportDataoperationsdefmy_run_function(operation_name,data_root_path,data_operations_file_path):project_data_ops=Dataoperations(path=data_root_path)project_data_ops.load_datasets(data_operations_file_path)project_data_ops.run(operation_name)and an example of the YAML data operations file that should be passed asdata_operations_file_path:My_process:operation_function:module:myprojectmodule.processname:my_functioninput_unit:type:DataInputFileUnitinput_path:'path/to/dataset.csv'pandas_read_function_name:read_csvsep:';'output_unit:type:DataOutputFileUnitoutput_path:'path/to/index_sorted_dataset.parquet'pandas_write_function_name:to_parquetMachine Learning partThis part is a humble contribution to accelerate Data Scientists recurrent operations, like for instance building lagged features for times series or exporting model features contribution. All the code produced here is Pandas and Scikit-learn friendly, as nearly everything is defined as a sklearn estimator.Tutorials 🎓Neural NetworksClassic Auto-encoders architecturesLSTM and CNN for text classification - Sentiment analysis applied to tweets in FrenchOutliers detection - Univariate time series and multivariate applicationSubpackages listSubpackageDescriptionCalibrationcalibration model techniquesEnsemblesome ensemble modelsExplainmodel explanation methodsFeature_engineeringsome categorical and time series pre-built feature engineeringFeature_selectionsome feature selection methodsMarkovsimple markov chains, supervised and unsupervised Hidden Markov ModelsNeural_networksend-to-end NN architecturesOutlierssome outlier detections techniques (MAD, FFT, Gaussian Process, etc.)Textsome NLP modelsVisualizationsome pre-built function to display common graphicsAbout copyrights on ML part: some code has been re-packaged from existing libraries which are not or fewly maintained, and for which I could have been involved in the past. All licences are respected, all original authors and repos are quoted.