Package: traceml
TraceMLEngine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.InstallpipinstalltracemlIf you would like to use the tracking features, you need to installpolyaxonas well:pipinstallpolyaxontraceml[WIP] Local sandboxComing soonOffline usageYou can enable the offline mode to track runs without an  API:exportPOLYAXON_OFFLINE="true"Or passing the offline flagfromtracemlimporttrackingtracking.init(...,is_offline=True,...)Simple usage in a Python scriptimportrandomimporttracemlastrackingtracking.init(is_offline=True,project='quick-start',name="my-new-run",description="trying TraceML",tags=["examples"],artifacts_path="path/to/artifacts/repo")# Tracking some data refstracking.log_data_ref(content=X_train,name='x_train')tracking.log_data_ref(content=y_train,name='y_train')# Tracking inputstracking.log_inputs(batch_size=64,dropout=0.2,learning_rate=0.001,optimizer="Adam")defget_loss(step):result=10/(step+1)noise=(random.random()-0.5)*0.5*resultreturnresult+noise# Track metricsforstepinrange(100):loss=get_loss(step)tracking.log_metrics(loss=loss,accuracy=(100-loss)/100.0,)# Track some one time resultstracking.log_outputs(validation_score=0.66)# Optionally manually stop the tracking processtracking.stop()Integration with deep learning and machine learning libraries and frameworksKerasYou can use TraceML's callback to automatically save all metrics and collect outputs and models, you can also track additional information using the logging methods:fromtracemlimporttrackingfromtraceml.integrations.kerasimportCallbacktracking.init(is_offline=True,project='tracking-project',name="keras-run",description="trying TraceML & Keras",tags=["examples"],artifacts_path="path/to/artifacts/repo")tracking.log_inputs(batch_size=64,dropout=0.2,learning_rate=0.001,optimizer="Adam")tracking.log_data_ref(content=x_train,name='x_train')tracking.log_data_ref(content=y_train,name='y_train')tracking.log_data_ref(content=x_test,name='x_test')tracking.log_data_ref(content=y_test,name='y_test')# ...model.fit(x_train,y_train,validation_data=(X_test,y_test),epochs=epochs,batch_size=100,callbacks=[Callback()],)PyTorchYou can log metrics, inputs, and outputs of Pytorch experiments using the tracking module:fromtracemlimporttrackingtracking.init(is_offline=True,project='tracking-project',name="pytorch-run",description="trying TraceML & PyTorch",tags=["examples"],artifacts_path="path/to/artifacts/repo")tracking.log_inputs(batch_size=64,dropout=0.2,learning_rate=0.001,optimizer="Adam")# Metricsforbatch_idx,(data,target)inenumerate(train_loader):output=model(data)loss=F.nll_loss(output,target)loss.backward()optimizer.step()tracking.log_metrics(loss=loss)asset_path=tracking.get_outputs_path('model.ckpt')torch.save(model.state_dict(),asset_path)# log modeltracking.log_artifact_ref(asset_path,framework="pytorch",...)TensorflowYou can log metrics, outputs, and models of Tensorflow experiments and distributed Tensorflow experiments using the tracking module:fromtracemlimporttrackingfromtraceml.integrations.tensorflowimportCallbacktracking.init(is_offline=True,project='tracking-project',name="tf-run",description="trying TraceML & Tensorflow",tags=["examples"],artifacts_path="path/to/artifacts/repo")tracking.log_inputs(batch_size=64,dropout=0.2,learning_rate=0.001,optimizer="Adam")# log modelestimator.train(hooks=[Callback(log_image=True,log_histo=True,log_tensor=True)])FastaiYou can log metrics, outputs, and models of Fastai experiments using the tracking module:fromtracemlimporttrackingfromtraceml.integrations.fastaiimportCallbacktracking.init(is_offline=True,project='tracking-project',name="fastai-run",description="trying TraceML & Fastai",tags=["examples"],artifacts_path="path/to/artifacts/repo")# Log model metricslearn.fit(...,cbs=[Callback()])Pytorch LightningYou can log metrics, outputs, and models of Pytorch Lightning experiments using the tracking module:fromtracemlimporttrackingfromtraceml.integrations.pytorch_lightningimportCallbacktracking.init(is_offline=True,project='tracking-project',name="pytorch-lightning-run",description="trying TraceML & Lightning",tags=["examples"],artifacts_path="path/to/artifacts/repo")...trainer=pl.Trainer(gpus=0,progress_bar_refresh_rate=20,max_epochs=2,logger=Callback(),)HuggingFaceYou can log metrics, outputs, and models of HuggingFace experiments using the tracking module:fromtracemlimporttrackingfromtraceml.integrations.hugging_faceimportCallbacktracking.init(is_offline=True,project='tracking-project',name="hg-run",description="trying TraceML & HuggingFace",tags=["examples"],artifacts_path="path/to/artifacts/repo")...trainer=Trainer(model=model,args=training_args,train_dataset=train_datasetiftraining_args.do_trainelseNone,eval_dataset=eval_datasetiftraining_args.do_evalelseNone,callbacks=[Callback],# ...)Tracking artifactsimportaltairasaltimportmatplotlib.pyplotaspltimportnumpyasnpimportplotly.expressaspxfrombokeh.plottingimportfigurefromvega_datasetsimportdatafromtracemlimporttrackingdefplot_mpl_figure(step):np.random.seed(19680801)data=np.random.randn(2,100)figure,axs=plt.subplots(2,2,figsize=(5,5))axs[0,0].hist(data[0])axs[1,0].scatter(data[0],data[1])axs[0,1].plot(data[0],data[1])axs[1,1].hist2d(data[0],data[1])tracking.log_mpl_image(figure,'mpl_image',step=step)deflog_bokeh(step):factors=["a","b","c","d","e","f","g","h"]x=[50,40,65,10,25,37,80,60]dot=figure(title="Categorical Dot Plot",tools="",toolbar_location=None,y_range=factors,x_range=[0,100])dot.segment(0,factors,x,factors,line_width=2,line_color="green",)dot.circle(x,factors,size=15,fill_color="orange",line_color="green",line_width=3,)factors=["foo 123","bar:0.2","baz-10"]x=["foo 123","foo 123","foo 123","bar:0.2","bar:0.2","bar:0.2","baz-10","baz-10","baz-10"]y=["foo 123","bar:0.2","baz-10","foo 123","bar:0.2","baz-10","foo 123","bar:0.2","baz-10"]colors=["#0B486B","#79BD9A","#CFF09E","#79BD9A","#0B486B","#79BD9A","#CFF09E","#79BD9A","#0B486B"]hm=figure(title="Categorical Heatmap",tools="hover",toolbar_location=None,x_range=factors,y_range=factors)hm.rect(x,y,color=colors,width=1,height=1)tracking.log_bokeh_chart(name='confusion-bokeh',figure=hm,step=step)deflog_altair(step):source=data.cars()brush=alt.selection(type='interval')points=alt.Chart(source).mark_point().encode(x='Horsepower:Q',y='Miles_per_Gallon:Q',color=alt.condition(brush,'Origin:N',alt.value('lightgray'))).add_selection(brush)bars=alt.Chart(source).mark_bar().encode(y='Origin:N',color='Origin:N',x='count(Origin):Q').transform_filter(brush)chart=points&barstracking.log_altair_chart(name='altair_chart',figure=chart,step=step)deflog_plotly(step):df=px.data.tips()fig=px.density_heatmap(df,x="total_bill",y="tip",facet_row="sex",facet_col="smoker")tracking.log_plotly_chart(name="2d-hist",figure=fig,step=step)plot_mpl_figure(100)log_bokeh(100)log_altair(100)log_plotly(100)Tracking DataFramesSummaryAn extension topandasdataframes describe function.The module containsDataFrameSummaryobject that extenddescribe()with:propertiesdfs.columns_stats: counts, uniques, missing, missing_perc, and type per columndsf.columns_types: a count of the types of columnsdfs[column]: more in depth summary of the columnfunctionsummary(): extends thedescribe()function with the values withcolumns_statsTheDataFrameSummaryexpect a pandasDataFrameto summarise.fromtraceml.summary.dfimportDataFrameSummarydfs=DataFrameSummary(df)getting the columns typesdfs.columns_typesnumeric9bool3categorical2unique1date1constant1dtype:int64getting the columns statsdfs.columns_statsABCDEcounts58025794578157814617uniques580235771128121missing0821211185missing_perc0%0.14%0.36%0.36%20.42%typesuniquecategoricalnumericnumericnumericgetting a single column summary, e.g. numerical column# we can also access the column using numbers A[1]dfs['A']std0.2827146max1.072792min0variance0.07992753mean0.55485165%0.160336725%0.319977650%0.496858875%0.827473295%1.011255iqr0.5074956kurtosis-1.208469skewness0.2679559sum3207.597mad0.2459508cv0.5095319zeros_num11zeros_perc0,1%deviating_of_mean21deviating_of_mean_perc0.36%deviating_of_median21deviating_of_median_perc0.36%top_correlations{u'D':0.702240243124,u'E':-0.663}counts5781uniques5771missing21missing_perc0.36%typesnumericName:A,dtype:object[WIP] SummariesAdd summary analysis between columns, i.e.dfs[[1, 2]][WIP] VisualizationsAdd summary visualization with matplotlib.Add summary visualization with plotly.Add summary visualization with altair.Add predefined profiling.[WIP] Catalog and VersionsAdd possibility to persist summary and link to a specific version.Integrate with quality libraries.