Package: pyemits
What is PyemitsPyEmits, a python package for easy manipulation in time-series data.The ultimate goal:Keep it simple and stupidMake everything configurableUniform API for machine learning and deep learningWhy need Pyemits?Time-series data is very common in real life.EngineeringFSI industry (Financial Services Industry)FMCG (Fast Moving Consumer Good)Data scientist's work consists of:forecastingprediction/simulationdata preparationcleansinganomaly detectiondescriptive data analysis/exploratory data analysis/data profiledata processing and ETL pipeline scriptseach new business unit shall build the following wheels again and againif you are facing these problems, then Pyemits is fit for youdata processing and ETL pipelineextractiontransformationcleansingfeature engineeringremove outliersAI landing for prediction, forecastingwrite it back to databaseml frameworkmultiple model trainingmultiple model predictionkfold validationanomaly detectionforecastingdevelop deep learning model (regression)ensemble modellingexploratory data analysisdescriptive data analysisdata profiledata set comparisondata scientist need to write different code to develop their model is there a package integrate all ml lib with single
simple api? That's why I create this project.This project is under active development, free to use (Apache 2.0)
I am happy to see anyone can contribute for more advancement on featuresNew feature:data processing pipelinedb connection and manipulationDevelopment ProgressFeaturesProgressAvailable at versionNotesPyOD integration80%0.1.2model parameters config are not yet finishedXGBoost integration80%0.1.2model parameters config are not yet finishedLightGBM integration80%0.1.2model parameters config are not yet finishedSklearn model integration80%0.1.2model parameters config are not yet finishedKeras integration100%0.1.2Pytorch_lightning integration100%0.1.2MXnet integration0%tbcDB connection0%tbcaggregation0%0.1.3cleansing0%0.1.3dimensional reduction0%0.1.3Kalman filtering0%0.1.3 or latermodel evaluation and visualization0%0.1.3 or laterdata profile for exploration20%0.1.3 or laterfinished data statistics onlyforecast at scale100%0.1.2see preprocessing.scaling.pyRelease UpdateVersionFeaturesNotes0.1initialization of project0.1.1RegTrainer/ParallelTrainer/KFoldCV0.1.2PyOD/Keras/Pytorch_lightning/scaling/splittingInstallpipinstallpyemitsFeatures highlightscikit-learn API styleinherit the design concept of pyecharts, "everything is configurable"highly flexible configuration items, can easily integrate with existing modeleasily integrate to SaaS product for product proof of conceptEasy trainingimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegTrainer,RegressionDataModelX=np.random.randint(1,100,size=(1000,10))y=np.random.randint(1,100,size=(1000,1))raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer(['XGBoost'],[None],raw_data_model)trainer.fit()Accept neural network as modelimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegTrainer,RegressionDataModelfrompyemits.core.ml.regression.nnimportKerasWrapperX=np.random.randint(1,100,size=(1000,10,10))y=np.random.randint(1,100,size=(1000,4))keras_lstm_model=KerasWrapper.from_simple_lstm_model((10,10),4)raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer([keras_lstm_model],[None],raw_data_model)trainer.fit()also keep flexibility on customized modelimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegTrainer,RegressionDataModelfrompyemits.core.ml.regression.nnimportKerasWrapperX=np.random.randint(1,100,size=(1000,10,10))y=np.random.randint(1,100,size=(1000,4))fromkeras.layersimportDense,Dropout,LSTMfromkerasimportSequentialmodel=Sequential()model.add(LSTM(128,activation='softmax',input_shape=(10,10),))model.add(Dropout(0.1))model.add(Dense(4))model.compile(loss='mse',optimizer='adam',metrics=['mse'])keras_lstm_model=KerasWrapper(model,nickname='LSTM')raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer([keras_lstm_model],[None],raw_data_model)trainer.fit()or attach it in algo configimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegTrainer,RegressionDataModelfrompyemits.core.ml.regression.nnimportKerasWrapperfrompyemits.common.config_modelimportKerasSequentialConfigX=np.random.randint(1,100,size=(1000,10,10))y=np.random.randint(1,100,size=(1000,4))fromkeras.layersimportDense,Dropout,LSTMfromkerasimportSequentialkeras_lstm_model=KerasWrapper(nickname='LSTM')config=KerasSequentialConfig(layer=[LSTM(128,activation='softmax',input_shape=(10,10),),Dropout(0.1),Dense(4)],compile=dict(loss='mse',optimizer='adam',metrics=['mse']))raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer([keras_lstm_model],[config],raw_data_model,{'fit_config':[dict(epochs=10,batch_size=32)]})trainer.fit()PyTorch, MXNet under development you can leave me a message if you want to contributeMultiOutput trainingimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegressionDataModel,MultiOutputRegTrainerfrompyemits.core.preprocessing.splittingimportSlidingWindowSplitterX=np.random.randint(1,100,size=(10000,1))y=np.random.randint(1,100,size=(10000,1))# when use auto-regressive like MultiOutput, pls set ravel = True# ravel = False, when you are using LSTM which support multiple dimensionsplitter=SlidingWindowSplitter(24,24,ravel=True)X,y=splitter.split(X,y)raw_data_model=RegressionDataModel(X,y)trainer=MultiOutputRegTrainer(['XGBoost'],[None],raw_data_model)trainer.fit()Parallel trainingprovide fast training using parallel jobuse RegTrainer as base, but add Parallel runningimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegressionDataModel,ParallelRegTrainerX=np.random.randint(1,100,size=(10000,1))y=np.random.randint(1,100,size=(10000,1))raw_data_model=RegressionDataModel(X,y)trainer=ParallelRegTrainer(['XGBoost','LightGBM'],[None,None],raw_data_model)trainer.fit()or you can use RegTrainer for multiple model, but it is not in Parallel jobimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegressionDataModel,RegTrainerX=np.random.randint(1,100,size=(10000,1))y=np.random.randint(1,100,size=(10000,1))raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer(['XGBoost','LightGBM'],[None,None],raw_data_model)trainer.fit()KFold trainingKFoldConfig is global config, will apply to allimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegressionDataModel,KFoldCVTrainerfrompyemits.common.config_modelimportKFoldConfigX=np.random.randint(1,100,size=(10000,1))y=np.random.randint(1,100,size=(10000,1))raw_data_model=RegressionDataModel(X,y)trainer=KFoldCVTrainer(['XGBoost','LightGBM'],[None,None],raw_data_model,{'kfold_config':KFoldConfig(n_splits=10)})trainer.fit()Easy predictionimportnumpyasnpfrompyemits.core.ml.regression.trainerimportRegressionDataModel,RegTrainerfrompyemits.core.ml.regression.predictorimportRegPredictorX=np.random.randint(1,100,size=(10000,1))y=np.random.randint(1,100,size=(10000,1))raw_data_model=RegressionDataModel(X,y)trainer=RegTrainer(['XGBoost','LightGBM'],[None,None],raw_data_model)trainer.fit()predictor=RegPredictor(trainer.clf_models,'RegTrainer')predictor.predict(RegressionDataModel(X))Forecast at scalesee examples:forecast at scale.ipynbData Modelfrompyemits.common.data_modelimportRegressionDataModelimportnumpyasnpX=np.random.randint(1,100,size=(1000,10,10))y=np.random.randint(1,100,size=(1000,1))data_model=RegressionDataModel(X,y)directly write an attribute to the data modeldata_model._update_attributes('X_shape',(1000,10,10))data_model.X_shape>>>(1000,10,10)write something to the meta datadata_model.add_meta_data('dimension',(1000,10,10))data_model.meta_data>>>{'dimension':(1000,10,10)}Anomaly detection (partial finished)see:anomaly detectionroot cause analyzer (under development)Kalman filter (under development)frompyemits.core.ml.anomaly_detection.predictorimportAnomalyPredictorfrompyemits.core.ml.anomaly_detection.trainerimportAnomalyTrainer,PyodWrapperfrompyemits.common.data_modelimportAnomalyDataModelfrompyemits.common.config_modelimportPyodIforestConfigfrompyod.models.iforestimportIForestfrompyod.utilsimportgenerate_datacontamination=0.1# percentage of outliersn_train=1000# number of training pointsn_test=200# number of testing pointsX_train,y_train,X_test,y_test=generate_data(n_train=n_train,n_test=n_test,contamination=contamination)# highly flexible model config, accept str, PyodWrapper with/without initialized model# either onetrainer=AnomalyTrainer(['IForest',PyodWrapper(IForest()),PyodWrapper(IForest),'IForest','IForest','IForest'],None,AnomalyDataModel(X_train))trainer=AnomalyTrainer([PyodWrapper(IForest(contamination=0.05)),PyodWrapper(IForest)],[None,PyodIforestConfig(contamination=0.05)],AnomalyDataModel(X_train))trainer.fit()# option 1predictor=AnomalyPredictor(trainer.clf_models)# option 2predictor=AnomalyPredictor(trainer.clf_models,other_config={'standard_scaler':predictor.misc_container['standard_scaler']})# option 3predictor=AnomalyPredictor(trainer.clf_models,other_config={'standard_scaler':predictor.misc_container['standard_scaler'],'combination_config':{'n_buckets':5}},combination_method='moa')predictor.predict(AnomalyDataModel(X_test))Data processing pipelineit features in the following:easy configurationregister steps, tasks in data processing pipelinelog data result in each tasks, each stepsrecord the flow of pipeline, from steps to work (from marco to micro)you can embed other function features in the task, but parameter: "data" is required to be passed ine.g. add email notification, add log, upload to database etc...frompyemits.core.preprocessing.pipelineimportDataNode,NumpyDataNode,PandasDataFrameDataNode,PandasSeriesDataNode,Pipeline,Step,Taskimportpandasaspdimportnumpyasnpdf=pd.DataFrame(np.random.random(size=(20,20)))dn=PandasDataFrameDataNode.from_pandas(df)defsum_each_col(data,a=1,b=2):returndata.sum()defsum_series(data):returnnp.array([data.sum()])task registration and arguments registrationtask_a=Task(sum_each_col)task_a.register_args(a=10,b=10)task_b=Task(sum_series)pipeline register step and executepipeline=Pipeline()step_a=Step('step_a',[task_a],'')step_b=Step('step_b',[task_b],'')pipeline.register_step(step_a)pipeline.register_step(step_b)pipeline.execute(dn)know the steps and its tasks from start to endpipeline.get_step_task_mapping()>>>{0:('test',['sum_each_col']),1:('test1',['sum_series'])}know the snapshot result in each steps, each tasks, friendly to data scientist for debuggingpipeline.get_pipeline_snapshot_res(step_id=1,tasks_id=0)>> > array([197.70351007])Evaluation (under development)see module:evaluationbacktestingmodel evaluationEnsemble (under development)blendingstackingvotingby combo packagemoaaomaveragemedianmaximizationIOdb connection and manipulationfrompyemits.common.io.dbimportDBConnectionBasedb=DBConnectionBase.from_full_db_path('sqlite:///test.db')db.execute('CREATE TABLE abcc(c1 int, c2 int, c3 int)')db.execute('INSERT INTO abcc(c1, c2, c3) VALUES (10, 10, 10)',change_commit=True)db.execute('SELECT * FROM abcc',fetch=10)db.execute('SELECT * FROM abcc',fetch='all')schemas=db.get_schemas()schemas['main']['abcc']>>[{'name':'c1','type':INTEGER(),'nullable':True,'default':None,'autoincrement':'auto','primary_key':0},{'name':'c2','type':INTEGER(),'nullable':True,'default':None,'autoincrement':'auto','primary_key':0},{'name':'c3','type':INTEGER(),'nullable':True,'default':None,'autoincrement':'auto','primary_key':0}]localdashboard ???other miscellaneous featurecontinuous evaluationaggregationdimensional reductiondata profile (intensive data overview)to be confirmed....Referencesthe following libraries gave me some idea/insightgreykitchangepoint detectionmodel summaryseaonalitypytorch-forecastingdartspyaforbitkats/prophets by facebooksktimegluon tstslearnpytsluminariestodsautotspyoddsscikit-hts