Package: liftpose
LiftPose3DLiftPose3D is a tool for transforming a 2D poses to 3D coordinates on labaratory animals. Classical approaches based on triangulation require synchronised acquisition from multiple cameras and elaborate calibration protocols. By contrast, LiftPose3D can reconstruct 3D poses from 2D poses from a single camera, in some instances without having to know the camera position or the type of lens used. For the theoretical background and details, have a look at ourpaper.To train LiftPose3D, ideally you would need (A) a 3D pose library, (B) corresponding 2D poses from the camera that you will use for lifting and (C) camera matrices (extrinsic and intrinsic).If you do not have access to(A) then use one of the provided datasets,(B) then obtain 2D images via projection using your camera matrices (you will need to calibrate to obtain these)(C) then place your camera further away to assume weak perspective.Starting-UpInstallationLiftPose3D PaperCiting LiftPose3DData formatDuring training, LiftPose3D accepts two numpy arrays in shape of[N J 2]and[N J 3]serving as input and output. Here, N is the number of poses and J is the number of joints. If you have multiple experiments, you can provide your data as dictionaries where the keys are strings and values are numpy arrays. You will also need the set at least one root joint and a set of target sets for each root joint. The network will predict the joints in the target sets relative to the root joints.For eachexample, we provide a uniqueload.pyfile to transform data into the required[N J 3]LifPose3D format.TrainingYou can train a network with the following generic syntax using experiment 1 for training and experiment 2 for testing.fromliftpose.mainimporttrainaslp3d_trainimportnumpy.random.randn_points,n_joints=100,5train_2d,test_2d=rand((n_points,n_joints,2)),rand((n_points,n_joints,2))train_3d,test_3d=rand((n_points,n_joints,3)),rand((n_points,n_joints,3))train_2d={"experiment_1":train_2d}train_3d={"experiment_1":train_3d}test_2d={"experiment_2":test_2d}test_3d={"experiment_2":test_3d}roots=[0]target_sets=[1,2,3,4]lp3d_train(train_2d,test_2d,train_3d,test_3d,roots,target_sets)By default, the outputs will be saved in a folderoutrelative to the path where LiftPose3D is called. You can change this behavior using theouput_folderparameter of thetrainfunction. You can take a look at thetrainfunction for otherdefault values and much longer documentation here.For example, you can further configure training by passing an extra argumenttraining_kwargsintrainfunction.fromliftpose.mainimporttrainaslp3d_traintraining_kwargs={"epochs":15,# train for 15 epochs"resume":True,# resume training where it was stopped"load":'ckpt_last.pth.tar'},# use last training checkpoint to resumelp3d_train(train_2d,test_2d,train_3d,test_3d,roots,target_sets,training_kwargs=training_kwargs)You can overwrite all the parameter insideliftpose.lifter.optusingtraining_kwargs.Training augmentationAugmenting training data is a great way to account for variability in the dataset, especially when training data is scarce.Currently, available options inliftpose.lifter.augmentationare:add_noise: adding Gaussian noise on 2D training pose to account for uncertainty in test timerandom_project: random projections of 3D pose in case camera orientation is unknown during test time (the training will ignore the inputtrain_2d)perturb_pose: pose augmentation by changing segment lengths when there are large animal-to-animal variationproject_to_cam: deterministic projections of 3D pose in case camera matrix is different and known in test timeTraining augmentation options can be specified in the argumentaugmentationand can be combined.fromliftpose.mainimporttrainaslp3d_trainfromliftpose.lifter.augmentationimportrandom_projectangle_aug={'eangles':{0:[[-10,10],[-10,10],[-10,10]]},#range of Euler angles (dictionary indexed by an integer which specifies the camera identify)'axsorder':'zyx',# order of rotations for euler angles'vis':None,# used in case not all joints are visible from a given camera'tvec':None,# camera translation vector'intr':None}# camera intrinsic matrixaug=[random_project(**angle_aug)]lp3d_train(train_2d,test_2d,train_3d,test_3d,roots,target_sets,aug=aug)See the case ofangle invariant Drosophila liftingfor an example implementation of augmentation.Inspecting the trainingThe training information is saved under thetrain_log.txt, which can be visualized as follows.fromliftpose.plotimportread_log_train,plot_log_trainepoch,lr,loss_train,loss_test,err_test=read_log_train(par['out_dir'])plot_log_train(plt.gca(),loss_train,loss_test,epoch)This will plot the training and test losses during the training.Testing the networkTo test the network on the data provided during thelp3d_traincall, runfromliftpose.mainimporttestaslp3d_testlp3d_test(par['out_dir'])Results will be saved inside thetest_results.pth.tarfile.To test the network in new data, runliftpose3d_test(par['out_dir'],test_2d,test_3d)where you provide thetest_2dandtest_3din the format described above. This will overwrite the previoustest_results.pklfile, if there is any.We also provide a simple interface for loading the test results from thetest_results.pklfile.fromliftpose.postprocessimportload_test_resultstest_3d_gt,test_3d_pred,_=load_test_results(par['out_dir'])This will return two numpy arrays:test_3d_gt, which is the same astest_3d, andtest_3d_pred, which has the predictions from the LiftPose3D.To generate the error distribution runfromliftpose.plotimportviolin_plotnames=['Head','Nose','Shoulder','Hip','Knee','Foot','Hand']violin_plot(plt.gca(),test_3d_gt=test_3d_gt,test_3d_pred=test_3d_pred,test_keypoints=np.ones_like(test_3d_gt),joints_name=names,units='m',body_length=2.21)Visualizing the 3D poseTo visualize the output 3D pose, first specify an animal skeleton in the fileparams.yaml. Note that bone information or the connected joints, are only used for visualization and not during training. You can have a closer look atplot_pose_3dfunction to see how the bone and color parameters are used during plotting.data:
    roots       : [0]
    target_sets : [[1, 2, 3, 4]]

vis:
    colors      : [[186, 30, 49]]
    bones       : [[0, 1], [1, 2], [2, 3], [3, 4]]
    limb_id     : [0, 0, 0, 0, 0]We provide the following function to visualize the 3D datafromliftpose.plotimportplot_pose_3dfig=plt.figure(figsize=plt.figaspect(1),dpi=100)ax=fig.add_subplot(111,projection='3d')ax.view_init(elev=-75,azim=-90)t=0plot_pose_3d(ax=ax,tar=test_3d_gt[t],pred=test_3d_pred[t],bones=par_data["vis"]["bones"],limb_id=par_data["vis"]["limb_id"],colors=par_data["vis"]["colors"],legend=True)This should output something similar to:You can also easily create moviesfromliftpose.plotimportplot_video_3dfig=plt.figure(figsize=plt.figaspect(1),dpi=100)ax=fig.add_subplot(111,projection='3d')ax.set_xlim([-4,2])ax.set_ylim([-3,2])ax.set_zlim([0,2])plot_video_3d(fig,ax,n=gt.shape[0],par=par_data,tar=gt,pred=pred,trailing=10,trailing_keypts=[4,9,14,19,24,29],fps=20)Usetrailingto plot pointstrailing_keyptswith a trailing effect with a desired length.Training with subset of pointsIn case you want to prevent some 2D/3D points from used in the training, you can passtrain_keyptsargument into thetrainfunction, which has the same shape astrain_3dbut has boolean datatype. Alternatively, in case you have missing keypoints, you can convert them tonp.NaN. In both cases, the loss from these points is not going to be used during backpropagation.