Package: tasrif
A Python framework for processing wearable data in the health domain.Tasrif is a library for processing of eHealth data. It provides:A pipeline DSL for chaining together commonly used processing operations on time-series eHealth
data, such as resampling, normalization, etc.DataReaders for reading eHealth datasets such asMyHeartCounts,SleepHealthand data from FitBit devices.InstallationTo use Tasrif, you will need to have the package installed. Please follow the bellow steps to install Tasrif:First, create a virtual environment usingvenvwith a linux operating system machine, or withWindows Subsystem for Linux# Create a virtual environmentpython3-mvenvtasrif-env# Activate the virtual environmentsourcetasrif-env/bin/activate# Upgrade pip(tasrif-env)pipinstall--upgradepipThen, install Tasrif either from PyPI(tasrif-env)pipinstalltasrifor install from source(tasrif-env)gitclonehttps://github.com/qcri/tasrif(tasrif-env)cdtasrif(tasrif-env)pipinstall-e.If no installation errors occur, seeQuick start by usecasesection to use Tasrif.Note on feature extraction using TasrifDue to some outdated internal Tasrif dependancies on Pypi, we have decided to place those dependancies inrequirements.txt. Once those packages are updated in Pypi, we will move them back tosetup.py. The currentrequirements.txtspecifies the dependancies links directly from Github. If you plan to use the following two operators:TSFreshFeatureExtractorOperatororCalculateTimeseriesPropertiesOperator, you will needTSFreshandKatspackages installed, which can be done by running the following command(tasrif-env)MINIMAL_KATS=1pipinstall-rrequirements.txtNote thatMINIMAL_KATS=1is passed in the installation script to minimally install Kats. Seerequirements.txtfor details.FeaturesPipeline DSLTasrif provies a variety of processing operators that can be chained together in a pipeline. The
operators themselves take as input and outputPandasDataFrames.For example, consider theAggregateOperator:>>>importpandasaspd>>>fromtasrif.processing_pipeline.customimportAggregateOperator>>>fromtasrif.processing_pipeline.pandasimportDropNAOperator>>>df0=pd.DataFrame([['Doha',25,30],['Doha',17,50],['Dubai',20,40],['Dubai',21,42]],columns=['city','min_temp','max_temp'])>>>operator=AggregateOperator(groupby_feature_names="city",aggregation_definition={"min_temp":["mean","std"]})>>>df0=operator.process(df0)>>>df0[citymin_temp_meanmin_temp_std0Doha21.05.6568541Dubai20.50.707107]Operators are meant to be used as part of a pipeline, where they can be chained together for
sequential processing of data:>>>importpandasaspd>>>fromtasrif.processing_pipelineimportSequenceOperator>>>fromtasrif.processing_pipeline.customimportAggregateOperator,CreateFeatureOperator>>>fromtasrif.processing_pipeline.pandasimportConvertToDatetimeOperator,SortOperator>>>df0=pd.DataFrame([['15-07-2021','Doha',25,30],['16-07-2021','Doha',17,50],['15-07-2021','Dubai',20,40],['16-07-2021','Dubai',21,42]],columns=['date','city','min_temp','max_temp'])>>>pipeline=SequenceOperator([ConvertToDatetimeOperator(feature_names=["date"]),CreateFeatureOperator(feature_name='avg_temp',feature_creator=lambdadf:(df['min_temp']+df['max_temp'])/2),SortOperator(by='avg_temp')])>>>pipeline.process(df0)[datecitymin_tempmax_tempavg_temp02021-07-15Doha253027.522021-07-15Dubai204030.032021-07-16Dubai214231.512021-07-16Doha175033.5]DataReadersTasrif also comes with DataReader classes for importing various eHealth datasets into pipelines.
These readers preprocess the raw data and convert them into a DataFrame for downstream processing in a pipeline.Supported datasets include:MyHeartCountsSleepHealthZenodo FitBitExport data from FitBit devicesExport data from Withings devices...and moreDataReaders can be used by treating them as source operators in a pipeline:fromtasrif.processing_pipelineimportSequenceOperatorfromtasrif.data_readers.my_heart_countsimportDayOneSurveyDatasetfromtasrif.processing_pipelineimportDropNAOperatorday_one_survey_path=<pathtoMyHeartCountsDayOneSurveyfile>pipeline=Pipeline([DayOneSurveyDataset(day_one_survey_path),DropNAOperator,SetIndexOperator('healthCode'),])pipeline.process()Quick start by usecaseReading dataCompute statisticsExtract features from existing columnsFilter dataWrangle dataTest prepared dataCreate a pipeline to link the operatorsDebug your pipelineDefine a custom operatorOther referencesReading dataReading a single csv filefromtasrif.processing_pipeline.pandasimportReadCsvOperatoroperator=ReadCsvOperator('examples/quick_start/csvs/participant1.csv')df=operator.process()[0]Reading multiple csvs in a folderfromtasrif.processing_pipeline.customimportReadCsvFolderOperatoroperator=ReadCsvFolderOperator(name_pattern='examples/quick_start/csvs/*.csv')df=operator.process()[0]by default,ReadCsvFolderOperatorconcatenates the csvs into one dataframe. if you would like to work on the csvs separately, you can pass the argumentconcatenate=FalsetoReadCsvFolderOperator, which returns a python generator that iterates the csvs.Reading csvs referenced by a column in dataframedfimportpandasaspdfromtasrif.processing_pipeline.customimportReadNestedCsvOperatordf=pd.DataFrame({"name":['Alfred','Roy'],"age":[43,32],"csv_files_column":['participant1.csv','participant2.csv']})operator=ReadNestedCsvOperator(folder_path='examples/quick_start/csvs/',field='csv_files_column')generator=operator.process(df)[0]forrecord,detailsingenerator:print(record)print(details)Reading json files referenced by a column in dataframedfimportpandasaspdfromtasrif.processing_pipeline.customimportIterateJsonOperatordf=pd.DataFrame({"name":['Alfred','Roy'],"age":[43,32],"json_files_column":['participant1.json','participant2.json']})operator=IterateJsonOperator(folder_path='examples/quick_start/jsons/',field='json_files_column',pipeline=None)generator=operator.process(df)[0]forrecord,detailsingenerator:print(record)print(details)Compute statisticsCompute quick statistics usingStatisticsOperator.StatisticsOperatorincludes counts of rows, missing data, duplicate rows, and others.importpandasaspdfromtasrif.processing_pipeline.customimportStatisticsOperatordf=pd.DataFrame([['2020-02-20',1000,1800,1],['2020-02-21',5000,2100,1],['2020-02-22',10000,2400,1],['2020-02-20',1000,1800,1],['2020-02-21',5000,2100,1],['2020-02-22',10000,2400,1],['2020-02-20',0,1600,2],['2020-02-21',4000,2000,2],['2020-02-22',11000,2400,2],['2020-02-20',None,2000,3],['2020-02-21',0,2700,3],['2020-02-22',15000,3100,3]],columns=['Day','Steps','Calories','PersonId'])filter_features={'Steps':lambdax:x>0}sop=StatisticsOperator(participant_identifier='PersonId',date_feature_name='Day',filter_features=filter_features)sop.process(df)[0]Or useParticipationOverviewOperatorto see statistics per participant. Pass the argumentoverview_type="date_vs_features"to compute statistics per date. See belowfromtasrif.processing_pipeline.customimportParticipationOverviewOperatorsop=ParticipationOverviewOperator(participant_identifier='PersonId',date_feature_name='Day',overview_type='participant_vs_features')sop.process(df)[0]UseAggregateOperatorif you require specific statistics for some columnsfromtasrif.processing_pipeline.customimportAggregateOperatoroperator=AggregateOperator(groupby_feature_names="PersonId",aggregation_definition={"Steps":["mean","std"],"Calories":["sum"]})operator.process(df)[0]Extract features from existing columnsConvert time columns into cyclical features, which are more efficiently grasped by machine learning modelsfromtasrif.processing_pipeline.customimportEncodeCyclicalFeaturesOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatordf=ReadCsvOperator('examples/quick_start/steps_per_day.csv',parse_dates=['Date']).process()[0]operator=EncodeCyclicalFeaturesOperator(date_feature_name="Date",category_definition="day")operator.process(df)[0]Extract timeseries features usingCalculateTimeseriesPropertiesOperatorwhich internally callskatspackagefromtasrif.processing_pipeline.katsimportCalculateTimeseriesPropertiesOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatordf=ReadCsvOperator('examples/quick_start/long_ts.csv',parse_dates=['Date']).process()[0]operator=CalculateTimeseriesPropertiesOperator(date_feature_name="Date",value_column='Steps')operator.process(df)[0]Extract using features usingtsfreshpackagefromtasrif.processing_pipeline.customimportSlidingWindowOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatorfromtasrif.processing_pipeline.tsfreshimportTSFreshFeatureExtractorOperatordf=ReadCsvOperator('examples/quick_start/cgm.csv',parse_dates=['dateTime']).process()[0]op=SlidingWindowOperator(winsize="1h15t",time_col="dateTime",label_col="CGM",participant_identifier="patientID")df_timeseries,df_labels,df_label_time,df_pids=op.process(df)[0]op=TSFreshFeatureExtractorOperator(seq_id_col="seq_id",date_feature_name='dateTime',value_col='CGM')features=op.process(df_timeseries)[0]features.dropna(axis=1)Note thatTSFreshFeatureExtractorOperatorrequires a columnseq_id. This column indicates which entities the time series belong to. Features will be extracted individually for each entity (id). The resulting feature matrix will contain one row per id. The column can be created manually or be created viaSlidingWindowOperator.Filter datafilter rows, days, or participants with a custom condition usingFilterOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatorfromtasrif.processing_pipeline.customimportFilterOperatordf=ReadCsvOperator('examples/quick_start/filter_example.csv',parse_dates=['Hours']).process()[0]operator=FilterOperator(participant_identifier="Id",date_feature_name="Hours",epoch_filter=lambdadf:df['Steps']>10,day_filter={"column":"Hours","filter":lambdax:x.count()<10,"consecutive_days":(7,12)# 7 minimum consecutive days, and 12 max},filter_type="include")operator.process(df)[0]Wrangle dataAdd a column usingCreateFeatureOperatorimportpandasaspdfrompandasimportTimestampdf=pd.DataFrame([[Timestamp('2016-12-31 00:00:00'),Timestamp('2017-01-01 09:03:00'),5470,2968,1],[Timestamp('2017-01-01 00:00:00'),Timestamp('2017-01-01 23:44:00'),9769,2073,1],[Timestamp('2017-01-02 00:00:00'),Timestamp('2017-01-02 16:54:00'),9444,2883,1],[Timestamp('2017-01-03 00:00:00'),Timestamp('2017-01-05 22:49:00'),20064,2287,1],[Timestamp('2017-01-04 00:00:00'),Timestamp('2017-01-06 07:27:00'),16771,2716,1]],columns=['startTime','endTime','steps','calories','personId'])operator=CreateFeatureOperator(feature_name="duration",feature_creator=lambdadf:df['endTime']-df['startTime'])operator.process(df)[0]Upsample or downsample date features usingResampleOperator. The first argumentrulecan be minutesmin, hoursH, daysD, and more. See details of resamplingherefromtasrif.processing_pipeline.pandasimportReadCsvOperatorfromtasrif.processing_pipeline.customimportResampleOperatordf=ReadCsvOperator('examples/quick_start/sleep.csv',parse_dates=['timestamp'],index_col=['timestamp']).process()[0]op=ResampleOperator('D',{'sleep_level':'mean'})op.process(df)Note that, currently, the index of the dataframe has to be of typeDatetimeIndexso thatResampleOperatorcan be called correctly.
Set the start hour of the day to some hour usingSetStartHourOfDayOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatorfromtasrif.processing_pipeline.customimportSetStartHourOfDayOperatordf=ReadCsvOperator('examples/quick_start/filter_example.csv',parse_dates=['Hours']).process()[0]operator=SetStartHourOfDayOperator(date_feature_name='Hours',participant_identifier='Id',shift=6)operator.process(df)[0]a new columnshifted_time_colwill be created. This can be useful if the user wants to calculate statistics at a redefined times of the day instead of midnight-to-midnight (e.g. 8:00 AM - 8:00 AM).Concatenate multiple dataframes or a generator usingConcatOperatorimportpandasaspdfromtasrif.processing_pipeline.pandasimportConcatOperatordf=pd.DataFrame([[Timestamp('2016-12-31 00:00:00'),Timestamp('2017-01-01 09:03:00'),5470,2968,1],[Timestamp('2017-01-01 00:00:00'),Timestamp('2017-01-01 23:44:00'),9769,2073,1],[Timestamp('2017-01-02 00:00:00'),Timestamp('2017-01-02 16:54:00'),9444,2883,1],[Timestamp('2017-01-03 00:00:00'),Timestamp('2017-01-05 22:49:00'),20064,2287,1],[Timestamp('2017-01-04 00:00:00'),Timestamp('2017-01-06 07:27:00'),16771,2716,1]],columns=['startTime','endTime','steps','calories','personId'])df1=df.copy()df2=df.copy()concatenated_df=ConcatOperator().process(df1,df2)[0]Normalize selected columnsimportpandasaspdfromtasrif.processing_pipeline.customimportNormalizeOperatorfromtasrif.processing_pipeline.customimportNormalizeTransformOperatordf=pd.DataFrame([[1,"2020-05-01 00:00:00",10],[1,"2020-05-01 01:00:00",15],[1,"2020-05-01 03:00:00",23],[2,"2020-05-02 00:00:00",17],[2,"2020-05-02 01:00:00",11]],columns=['logId','timestamp','sleep_level'])op=NormalizeOperator('all','minmax',{'feature_range':(0,2)})output=op.process(df)Use the fit normalizer on different data usingNormalizeTransformOperatortrained_model=output[0][1]op=NormalizeTransformOperator('all',trained_model)output=op.process(df)outputUseAggregateActivityDatesOperatorto view the start date and end date of a
dataframe that has a date column per row per participant.importpandasaspdfromtasrif.processing_pipeline.customimportAggregateActivityDatesOperatorfromtasrif.processing_pipeline.pandasimportReadCsvOperatorreader=ReadCsvOperator('examples/quick_start/activity_long.csv')df=reader.process()[0]operator=AggregateActivityDatesOperator(date_feature_name="date",participant_identifier=['Id','logId'])df=operator.process(df)[0]dfYou can usejqOperatorto process JSON dataimportpandasaspdfromtasrif.processing_pipeline.customimportJqOperatordf=[{"date":"2020-01-01","sleep":[{"sleep_data":[{"level":"rem","minutes":180},{"level":"deep","minutes":80},{"level":"light","minutes":300}]}]},{"date":"2020-01-02","sleep":[{"sleep_data":[{"level":"rem","minutes":280},{"level":"deep","minutes":60},{"level":"light","minutes":200}]}]}]op=JqOperator("map({date, sleep: .sleep[].sleep_data})")op.process(df)Test prepared dataSee if your prepared data can act as an input to a machine learning modelfromtasrif.processing_pipeline.customimportLinearFitOperatordf=pd.DataFrame([[1,"2020-05-01 00:00:00",10,'poor'],[1,"2020-05-01 01:00:00",15,'poor'],[1,"2020-05-01 03:00:00",23,'good'],[2,"2020-05-02 00:00:00",17,'good'],[2,"2020-05-02 01:00:00",11,'poor']],columns=['logId','timestamp','sleep_level','sleep_quality'])op=LinearFitOperator(feature_names='sleep_level',target='sleep_quality',target_type='categorical')op.process(df)Create a pipeline to link the operatorsChain operators usingSequenceOperatorimportpandasaspdfromtasrif.processing_pipelineimportSequenceOperatorfromtasrif.processing_pipeline.customimportAggregateOperator,CreateFeatureOperator,SetStartHourOfDayOperatorfromtasrif.processing_pipeline.pandasimportConvertToDatetimeOperator,SortOperator,ReadCsvOperatordf=ReadCsvOperator('examples/quick_start/cgm.csv').process()[0]dfpipeline=SequenceOperator([ConvertToDatetimeOperator(feature_names=["dateTime"]),SetStartHourOfDayOperator(date_feature_name='dateTime',participant_identifier='patientID',shift=6),SortOperator(by='dateTime'),AggregateOperator(groupby_feature_names="patientID",aggregation_definition={"CGM":["mean","std"]})])pipeline.process(df)Debug your pipelineTasrif contains observers undertasrif/processing_pipeline/observers/that are useful for seeing how the operators change your data. For instance, you can print the head of processed dataframe after every operator. You can do so by passing anobserverto theobserversargument inSequenceOperator.importpandasaspdfromtasrif.processing_pipeline.pandasimportRenameOperatorfromtasrif.processing_pipeline.observersimportFunctionalObserver,LoggingObserver,GroupbyLoggingObserverfromtasrif.processing_pipelineimportSequenceOperator,Observerdf=pd.DataFrame([[1,"2020-05-01 00:00:00",1],[1,"2020-05-01 01:00:00",1],[1,"2020-05-01 03:00:00",2],[2,"2020-05-02 00:00:00",1],[2,"2020-05-02 01:00:00",1]],columns=['logId','timestamp','sleep_level'])pipeline=SequenceOperator([RenameOperator(columns={"timestamp":"time"}),RenameOperator(columns={"time":"time_difference"})],observers=[LoggingObserver("head,tail")])result=pipeline.process(df[0])resultDefine a custom operatorUsers can inherit fromMapProcessingOperatorto quickly build their own custom operators that perform map-like operations.fromtasrif.processing_pipeline.map_processing_operatorimportMapProcessingOperatorclassSizeOperator(MapProcessingOperator):def_processing_function(self,df):returndf.sizeOther referencesYou may examinetasrif/processing_pipeline/test_scripts/for other minimal examples of Tasrif's operators.Common Pandas functions can be found undertasrif/processing_pipeline/pandas/DocumentationTasrif's official documentation is hosted here:https://tasrif.qcri.orgYou can build the docs locally after installing the dependencies insetup.pyandrequirements.txtby:cd docs
make htmlYou can then browse through them by openingdocs/build/html/index.htmlin a browser.ContributingThis project is much stronger with your collaboration. Be part of it!Thank you all amazing contributors!