Package: bapipe-keypoints-telfer
Behaviour Analysis for Keypoint DataA pipeline for turning keypoint data gathered from deep learning models into data and visualizations.BackgroundAdvances in Deep Learning have driven a wave of pose-estimation tools which extract information from animals and their surroundings (DeepLabCut,OpenPose,SLEAP). These models are trained to extract keypoint data (x, y coordinates) for specified bodyparts or objects in the environment.This pipeline automates a number of steps involved in turning this keypoint data from behavioural experiments into meaningful data and visuallizations. Some of these steps includeAligning videos to account for changes in camera perspectiveSeveral outlier-filtering strategiesCreating videos and other validation graphsTrimming video duration for when the mouse is first visibleWe demonstrate a few examples of using DeepLabCut data in order to produceCustom validation videosTotal distance travelled by treatment groupHeatmaps by treatment group treatment groupsTime spent in a zoneEntire experiments can often be processed quite quickly at this level, so this pipeline enables the ability to evaluate experiments for multiple parameters (e.g. different zone sizes) and develop real time dashboards for analysis. Furthermore, it includes an image registration step that removes scale and position variance between videos, allowing for perfectly aligned analysis not available in existing commercial or open source tools.Author: Andre Telfer (andretelfer@cmail.carleton.ca)Installationpipinstall-U-qbapipe-keypoints-telferNote: you may need to restart the kernel to use updated packages.To install the local codepipinstall--upgrade--no-deps--force-reinstall..DockerfilesFor convenience, we also provide dockerfiles to replicate our analysis environmentgitclonegit@github.com:A-Telfer/bapipe-keypoints.gitcdbapipe-keypoints
bashdocker-run.shStructure of DataIn order to use this pipeline, load a csv file with the following informationsubject idpath to videopath to mouse bodypart position files (produced from DeepLabCut or similar pose estimation software)path to video landmarks (e.g. box corners) [optional]path to camera calibrations [optional]%%timeimportpandasaspdfrompathlibimportPathPROJECT=Path("/home/jovyan/shared/shared/curated/fran/v2")datafiles=pd.read_csv(PROJECT/'datafiles.csv')datafiles.head(3)CPU times: user 378 ms, sys: 1.03 s, total: 1.41 s
Wall time: 210 msidvideomouse_labelslandmark_labelscamera_calibrations0m1videos/m1.mp4videos/mouse_labels/m1DLC_resnet50_agrpNov19sh...videos/landmark_labels/m1DLC_resnet50_agrp_lan...camera_calibrations.json1m2videos/m2.mp4videos/mouse_labels/m2DLC_resnet50_agrpNov19sh...videos/landmark_labels/m2DLC_resnet50_agrp_lan...camera_calibrations.json2m3videos/m3.mp4videos/mouse_labels/m3DLC_resnet50_agrpNov19sh...videos/landmark_labels/m3DLC_resnet50_agrp_lan...camera_calibrations.jsonLoad ExperimentNormalize videosRegister arena cornersRemove lens warpingClip start time for when mouse is first visibleOutlier correctionRemove based on pairwise distancesRemove based on bodypart velocitiesProvide an api for common operationsParallelized analysis to reduce run times by several timesProvides visibility for common features%%timeimportnumpyasnpimportbapipeconfig=bapipe.AnalysisConfig(box_shape=(400,300),# size of the box in mm (or any other units)remove_lens_distortion=True,# remove distortion caused by camera lens (requires a calibration file)use_box_reference=True,# align all of the videos for the test arena)video_set=bapipe.VideoSet.load(datafiles,config,root_dir=PROJECT)video_set100%|████████████████████████████████████████████████████████| 72/72 [00:09<00:00,  7.63it/s]


CPU times: user 1.83 s, sys: 737 ms, total: 2.57 s
Wall time: 10.1 sNumber of videos72Video sizes (W,H)960x540Total duration [s]47837.6Average duration [s]664.41Compare original videos to corrected videos%%timeimportmatplotlib.pyplotaspltgs=plt.GridSpec(1,100)plt.figure(figsize=(16,10))plt.subplot(gs[:52])plt.title("Original")override_config={'use_box_reference':False,'remove_lens_distortion':False}plt.imshow(bapipe.create_video_grid(video_set,override_config=override_config))plt.axis('off')plt.subplot(gs[60:])plt.title("Aligned")plt.imshow(bapipe.create_video_grid(video_set))plt.axis('off')plt.show()CPU times: user 45.3 s, sys: 5.51 s, total: 50.9 s
Wall time: 8.9 sAnalysis Examples%%timetreatment_data=pd.read_csv(PROJECT/'cohorts1&2.csv',index_col='animal')treatment_data.head(3)CPU times: user 3.49 ms, sys: 0 ns, total: 3.49 ms
Wall time: 2.65 ms<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>injected_oninjected_withamount_eatentreatment1treatment2cohortlatency_to_approachtime_in_cornertime_eatingsexanimalm12021-07-16saline/saline0.0salinesaline16.18245.1716.15malem22021-07-15saline/ghrelin0.2salineghrelin122.97124.2135.81malem32021-07-15mt2/ghrelin0.1mt2ghrelin110.39130.0911.01maleExample 1: Video ValidationAnnotate what's being scored over the videos%%timefromtqdmimporttqdmfromIPython.displayimportVideovideo=video_set[0]withbapipe.VideoWriter(video,'test.mp4')aswriter:foriintqdm(range(1000,1100)):frame=video.get_frame(i)bapipe.draw_dataframe_points(frame,video.mouse_df,i)writer.write(frame)Video('test.mp4')100%|██████████████████████████████████████████████████████| 100/100 [00:04<00:00, 22.82it/s]

CPU times: user 24.6 s, sys: 3.22 s, total: 27.8 s
Wall time: 4.46 sYour browser does not support thevideoelement.Example 2: Distance Travelled%%timeimportseabornassnsdefget_distance_travelled(video):# Average the bodypart positions of the mouse to get its centroidcentroid=video.mouse_df.groupby(level='coords',axis=1).mean()# Get the between-frame movement of the bodypartsdeltas=centroid[['x','y']].diff().dropna()# Calculate the total distance travelledreturnnp.sum(np.linalg.norm(deltas.values,axis=1))distances=pd.Series(video_set.apply(get_distance_travelled),index=video_set.index,name='distance')sns.barplot(data=treatment_data.join(distances),x='injected_with',y='distance')plt.xlabel("Treatment Group")plt.ylabel("Distance Travelled [mm]")plt.title("Locomotivity: Distance Travelled")plt.show()100%|███████████████████████████████████████████████████████| 72/72 [00:00<00:00, 216.58it/s]CPU times: user 439 ms, sys: 977 ms, total: 1.42 s
Wall time: 660 msExample 2: HeatmapsShow what zones the mice are spending their timeFind position density%%timefromtqdmimporttqdmfromscipy.stats.kdeimportgaussian_kdegroups=treatment_data.groupby(['treatment1','treatment2'])w,h=config.box_shaperesult={}foridx,groupintqdm(groups):group_videos=[video_set[video_set.index.index(idx)]foridxingroup.index]# Stack the mouse-location dataframes for each mouse in this treatment groupgroup_df=pd.concat([video.mouse_dfforvideoingroup_videos],axis=0).dropna()# Get the centroid of the mouse by averaging the bodypart positions in each framecentroid=group_df.groupby(level='coords',axis=1).mean()[['x','y']]data=centroid[['y','x']].values.T# Get the density of time spent in location (down sampled for 1 frame every 100)k=gaussian_kde(data[:,::100],)mgrid=np.mgrid[:h,:w]z=k(mgrid.reshape(2,-1))result['/'.join(idx)]=z<timed exec>:2: DeprecationWarning: Please use `gaussian_kde` from the `scipy.stats` namespace, the `scipy.stats.kde` namespace is deprecated.
100%|██████████████████████████████████████████████████████████| 4/4 [00:11<00:00,  2.86s/it]

CPU times: user 12.3 s, sys: 4.22 s, total: 16.5 s
Wall time: 11.4 sCreate contour plots%%timeimportmatplotlib.pyplotaspltCONTOUR_LEVELS=20video=video_set[0]frame=video.get_frame(1000)plt.figure(figsize=(20,5))gs=plt.GridSpec(1,5)plt.title("Box Reference")plt.subplot(gs[0])plt.imshow(frame)foridx,(gname,z)inenumerate(result.items()):plt.subplot(gs[idx+1])plt.title(gname)plt.imshow(frame)# plotting a frame sets up the matplotlib axis correctlyplt.imshow(z.reshape(mgrid.shape[1:]))plt.contourf(z.reshape(mgrid.shape[1:]),cmap='seismic',alpha=1,levels=CONTOUR_LEVELS)CPU times: user 589 ms, sys: 50 ms, total: 639 ms
Wall time: 178 msExample 3: Zone based analysisZones on can be drawn once in tools like napari, or automatically plottedIf the videos have been normalized, zones can be drawn once and applied to all videos%%timedefget_center_zone(video):w,h=video_set[0].config.box_shapecx,cy=w//2,h//2s=50returnplt.Polygon([[cx-s,cy-s],[cx-s,cy+s],[cx+s,cy+s],[cx+s,cy-s]],alpha=0.5,label='Center zone')deftime_in_zone(video):center_zone=get_center_zone(video)centroid=video.mouse_df.groupby(level='coords',axis=1).mean()[['x','y']].valuesreturnnp.sum(center_zone.contains_points(centroid))/video.fpsdata=pd.Series(video_set.apply(time_in_zone),index=video_set.index,name='time_in_center_zone')plt.figure(figsize=(15,5))gs=plt.GridSpec(1,2)plt.subplot(gs[0])v=video_set[0]plt.imshow(v.get_frame(900))plt.gca().add_patch(get_center_zone(v))plt.subplot(gs[1])sns.barplot(data=treatment_data.join(data),x='injected_with',y='time_in_center_zone')plt.xlabel("Treatment Group")plt.ylabel("Time in Zone [s]")plt.title("Time in Zone")100%|███████████████████████████████████████████████████████| 72/72 [00:00<00:00, 185.93it/s]


CPU times: user 583 ms, sys: 309 ms, total: 892 ms
Wall time: 682 ms





Text(0.5, 1.0, 'Time in Zone')