Package: nely-df3d
Markerless Multi-view Motion Capture for Tethered DrosophilaDeepFly3D is a PyTorch and PyQT5 implementation of 2D-3D tethered Drosophila pose estimation. It aims to provide an interface for pose estimation and to permit further correction of 2D pose estimates, which are automatically converted to 3D pose.DeepFly3D does not require a calibration pattern, it enforces geometric constraints using pictorial structures, which corrects most of the errors, and the remaining errors are automatically detected can be dealt manually with GUI.We previously published our DeepFly3D work on eLife journal. You can read the publicationhere.Table of ContentsInstallingInstalling with pipOdd CUDA DriversInstalling from the sourceData StructureBasic UsageAdvanced UsagePython InterfaceVideosOutputPoints2dCamera OrderingHeatmap ConfidenceRunning GUIVersion HistoryInstallingInstalling with pipCreate a new anaconda environment, and pip install nely-df3d package.condacreate-ndf3dpython=3.6
condaactivatedf3d
pipinstallnely-df3dOdd CUDA DriversOnly in case your cuda driver is not up-to-date, or is not supported by mainstream pytorch, additionally you might need to explicitly install cudatoolkit before pip installing nely-df3d:condainstallpytorchtorchvisiontorchaudiocudatoolkit="YOUR_CUDA_VERSION"-cpytorchFor example with with RTX 3080 Ti GPU, you will need to do:condacreate-ndf3dpython=3.6
condaactivatedf3d
condainstallpytorchtorchvisioncudatoolkit=11-cpytorch-nightly
pipinstallnely-df3dInstalling from the sourceDeepFly3D requires Python3, Anaconda environment and CUDA drivers for installation. It isonly tested on Ubuntu and MacOS. First, clone the repository:git clone https://github.com/NeLy-EPFL/DeepFly3D
cd DeepFly3DThen, run create a conda environment withconda create -n df3d python=3.6which will create a new python environment. Then, activate the environment.conda activate df3dOnce this is done  you can install thedf3dpackage with the following command,pip install -e .which uses the setup.py function to create the package.Make sure you also have installed the CUDA drivers compatible with your GPU, otherwise it is not possible to make 2D predictions. You can check how to install CUDA drivers here:https://developer.nvidia.com/cuda-downloadsData StructureThe inteded usage of DeepFly3D is through command-line-intarface (CLI).
df3d-cli assumes there are videos or images in this format under the folder.
if your path /your/image/path has images or videos, df3d-cli will run 2D pose estimation, calibration and triangulation and will save 2d pose, 3d pose and calibration parameters under the folder /your/image/path/df3d.Idealy you would have images or videos underimages/folder, with the specific naming convention:.
+-- images/
|   +-- camera_0_img_0.jpg
|   +-- camera_1_img_0.jpg
|   +-- camera_2_img_0.jpg
|   +-- camera_3_img_0.jpg
|   +-- camera_4_img_0.jpg
|   +-- camera_5_img_0.jpg
|   +-- camera_6_img_0.jpg
...or.
+-- images
|   +-- camera_0.mp4
|   +-- camera_1.mp4
|   +-- camera_2.mp4
|   +-- camera_3.mp4
|   +-- camera_4.mp4
|   +-- camera_5.mp4
|   +-- camera_6.mp4In case of mp4 files, df3d will first expand them into images using ffmpeg. Please check the sample data for a real exampe:https://github.com/NeLy-EPFL/DeepFly3D/tree/master/sample/testBasic UsageThe basic usage is like this.df3d-cli/your/image/path\--order0123456camera order stands for the selection of cameras. The default camera ordering (0 1 2 3 4 5 6) stands for this. In case you have some other order, then you need to  tell which order.Originally.Then if you have the following order, yourSo for example, if your data looks like this, then your order should be 6 5 4 3 2 1 0.Advanced Usageusage:df3d-cli[-h][-v][-vv][-d][--output-folderOUTPUT_FOLDER][-r][-f][-o][-nNUM_IMAGES_MAX][-order[CAMERA_IDS[CAMERA_IDS...]]][--video-2d][--video-3d][--skip-pose-estimation]INPUT

DeepFly3Dposeestimation

positionalarguments:INPUTWithoutadditionalarguments,afoldercontainingunlabeledimages.

optionalarguments:-h,--helpshowthishelpmessageandexit-v,--verboseEnableinfooutput(suchasprogressbars)-vv,--verbose2Enabledebugoutput-d,--debugDisplaystheargumentlistfordebuggingpurposes--output-folderOUTPUT_FOLDERThenameofsubfolderwheretowriteresults-r,--recursiveINPUTisafolder.Successivelyuseitssubfoldersnamed'images/'-f,--from-fileINPUTisatext-file,whereeachlinenamesafolder.Successivelyusethelistedfolders.-o,--overwriteRerunposeestimationandoverwriteexistingposeresults-nNUM_IMAGES_MAX,--num-images-maxNUM_IMAGES_MAXMaximalnumberofimagestoprocess.-order[CAMERA_IDS[CAMERA_IDS...]],--camera-ids[CAMERA_IDS[CAMERA_IDS...]]Orderingofthecamerasprovidedasanorderedlistofids.Example:0143256.--video-2dGeneratepose2dvideos--video-3dGeneratepose3dvideos--skip-pose-estimationSkip2Dand3DposeestimationTherefore, you can create advanced queries in df3d-cli, for example:df3d-cli-f/path/to/text.txt\# process each line from the text file-r\# recursively search for images folder under each line of the text line--order0123456\# set the camera order-n100\# process only the first 100 images--output-folderresults\# write results under  /your/image/path/results instead of  /your/image/path/df3d--vv\# will print agressivelly, for debugging purposes--skip-pose-estimation\# will not run 2d pose estimation, instead will do calibration, triangulation and will save results--video-2d\# will make 2d video for each folder--video-3d\# will make 3d video for each folderTo test df3d-cli, you run it on a folder for only 100 images, make videos, and print agressivelly for debugging:df3d-cli/path/to/images/-n100-vv-order0123456Python InterfaceOptionally, you can also use df3d on directly python.fromdf3d.coreimportCorefromdf3dimportvideocore=Core(input_folder='../sample/test/',num_images_max=100,output_subfolder='df3d_py',camera_ordering=[0,1,2,3,4,5,6])core.pose2d_estimation()core.calibrate_calc(min_img_id=0,max_img_id=100)# save df3d_resultt  file under '../sample/test/df3d_py'core.save()# make videosvideo.make_pose2d_video(core.plot_2d,core.num_images,core.input_folder,core.output_folder)video.make_pose3d_video(core.get_points3d(),core.plot_2d,core.num_images,core.input_folder,core.output_folder,)In general following functions are available for Core module:classCore:def__init__(self,input_folder,num_images_max):# 9 linesdefsetup_cameras(self):# 38 lines# attribute access@propertydefinput_folder(self):# 2 lines@propertydefoutput_folder(self):# 2 lines@propertydefimage_shape(self):# 2 lines@propertydefnumber_of_joints(self):# 3 linesdefhas_pose(self):# 1 linesdefhas_heatmap(self):# 1 linesdefhas_calibration(self):# 4 lines# interactions with pose-estimationdefupdate_camera_ordering(self,cidread2cid):# 12 linesdefpose2d_estimation(self):# 14 linesdefnext_error(self,img_id):# 1 linesdefprev_error(self,img_id):# 1 linesdefcalibrate_calc(self,min_img_id,max_img_id):# 35 linesdefnearest_joint(self,cam_id,img_id,x,y):# 10 linesdefmove_joint(self,cam_id,img_id,joint_id,x,y):# 10 linesdefsave_calibration(self):# 3 linesdefsave_pose(self):# 63 linesdefsave_corrections(self):# 1 line# visualizationsdefplot_2d(self,cam_id,img_id,with_corrections=False,joints=[]):# 33 linesdefplot_heatmap(self,cam_id,img_id,joints=[]):# 5 linesdefget_image(self,cam_id,img_id):# 4 lines# private helper methodsdefnext_error_in_range(self,range_of_ids):# 6 linesdefget_joint_reprojection_error(self,img_id,joint_id,camNet):# 11 linesdefjoint_has_error(self,img_id,joint_id):# 4 linesdefsolve_bp_for_camnet(self,img_id,camNet):# 29 linesVideosUsing the flag --video-2d with df3d-cli will create the following video:Using the flag --video-3d with df3d-cli will create the following video:Outputdf3d-cli saves results under df3d_result.pk file. You can read it using,result_path='../sample/test/df3d/df3d_result*.pkl'd=pickle.load(open(glob.glob(pr_path)[0],'rb'))This will read a dictionary with the following keys:d.keys()>>>dict_keys([0,1,2,3,4,5,6,'points3d','points2d','points3d_wo_procrustes','camera_ordering','heatmap_confidence'])Points2DDetected 2D keypoints are hold under d['points2d'], which is a 4 dimensional tensor.d['points2d'].shape>>>(7,15,38,2)# [CAMERAS, TIMES, JOINTS, 2D]You can read the corresponding 2D points from a particular camera from a particular time using,row,column=d['points2d'][CAMERA,TIME,JOINT]The points are in the (row, column) format.You can also visualize which keypoints in results belongs to which keypoints on the animal:importmatplotlib.pyplotaspltimage_path='../sample/test/camera_{cam_id}_img_{img_id}.jpg'pr_path='../sample/test/df3d/df3d_result*.pkl'cam_id,time=0,0plt.imshow(plt.imread(image_path.format(cam_id=0,img_id=0)))plt.axis('off')forjoint_idinrange(19):x,y=d['points2d'][cam_id,time][joint_id,1]*960,d['points2d'][cam_id,time][joint_id,0]*480plt.scatter(x,y,c='blue',s=5)plt.text(x,y,f'{i}',c='red')Points3DYou can recalculate the 3D points, given the 2D points and the caibraiton parameters:frompyba.CameraNetworkimportCameraNetworkimportpickleimportglobimage_path='./sample/test/camera_{cam_id}_img_{img_id}.jpg'pr_path='./sample/test/df3d/df3d_result*.pkl'd=pickle.load(open(glob.glob(pr_path)[0],'rb'))points2d=d['points2d']# df3d points2d are saved in normalized into [0,1], rescale them into image shapecamNet=CameraNetwork(points2d=points2d*[480,960],calib=d,image_path=image_path)points3d=camNet.triangulate()Camera 0 corresponds to origin. It's camera center (not the translation vector) corresponds to 0 point.Camera OrderingThe same camera ordering as given input using --order flag in cli.d["camera_ordering"]>>>array([0,1,2,3,4,5,6])Heatmap ConfidenceStacked Hourglass confidence values for each joint predicted. Given an unnormalized posterior distribution heatmap H over the pixels, we take the argmax_{h, w} H for the final prediction and H[h, w] for the confidence level.Calibrationdf3d_result files also have the calculated calibration parameters for each camera. Each calibration section includesrotation matrix Rtranslation vector tvec,intrinsic matrix intr,distortion parameters distort.calib={0:{'R':array([[0.90885957,0.006461,-0.41705219],[0.01010426,0.99924554,0.03750006],[0.41697983,-0.0382963,0.90810859]]),'tvec':array([1.65191596e+00,2.22582670e-02,1.18353733e+02]),'intr':array([[1.60410e+04,0.00000e+00,2.40000e+02],[0.00000e+00,1.59717e+04,4.80000e+02],[0.00000e+00,0.00000e+00,1.00000e+00]]),'distort':array([0.,0.,0.,0.,0.])},1:{'R':array([[0.59137248,0.02689833,-0.80594979],[-0.00894927,0.9996009,0.02679478],[0.80634887,-0.00863303,0.59137718]]),'tvec':array([1.02706542e+00,-9.25820468e-02,1.18251732e+02]),'intr':array([[1.60410e+04,0.00000e+00,2.40000e+02],[0.00000e+00,1.59717e+04,4.80000e+02],[0.00000e+00,0.00000e+00,1.00000e+00]]),'distort':array([0.,0.,0.,0.,0.])},}The coordinate system is compatible with OpenCV, where z-axis corresponds to axis going out of camera.Running GUIGUI is primarily used for correcting the false 2D pose estimation results in the 'Correction' mode. Your changes will be saved under df3d folder and will be used for the final df3d_result file.Currently, you can only use GUI after running the df3d on the cli on the same folder.After installing the dependencies we can initialize the GUI using the command line entry point:df3d ./data/test/ 15The second argument sets the image folder, while the third argument sets the upper bound for the images, in case you only want to process the subset of images.This should start the GUI:you can optionally remove/FULL/PATH_FOLDERandNUM_IMAGES, in which case pop-up apperas the select the folder.After completing pose estimation in the cli, you can open the pose mode:DevelopmentDeepFly3D consists of 3 pip packages:DeepFly3D:https://pypi.org/project/df3d/PyBundleAdjustment:https://pypi.org/project/pyba/Drosophila2D Pose:https://pypi.org/project/df2d/The master branch of the DeepFly3D package is kept up-to-date with the last version of the pip package. Development is done under dev branch. Before pushing changes to the master branch, make sure all test cases are passing. You can run the tests using python test.py. Unittests make sure several scenarios can be processed using cli without failing.ReferencesYou can cite our paper in case you find it useful.@inproceedings{Gunel19DeepFly3D,
  author    = {Semih G{\"u}nel and
               Helge Rhodin and
               Daniel Morales and
               João Compagnolo and
               Pavan Ramdya and
               Pascal Fua},
  title     = {DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila},
  bookTitle = {eLife},
  doi       = {10.7554/eLife.48571},
  year      = {2019}
}Version HistoryChanges in 0.5Major internal rewrite.Changes in 0.4Using the CLI, the output folder can be changed using the--output-folderflagCLI and GUI now use the same pose estimation code, so changes will automatically propagate to bothMinor tweaks in the GUI layout, functionality kept unchangedChanges in 0.3Results are saved in df3d folder instead of the image folder.Much faster startup time.Cameras are automatically ordered using Regular Expressions.CLI improvements. Now it includes 3D pose.Changes in 0.2Changing name from deepfly3d to df3dAdding cli interface with df3d-cliRemoving specific dependencies for numpy and scipyRemoving L/R buttons, so you can see all the data at onceRemoving the front cameraFaster startup time, less time spent on searching for the image folderBetter notebooks for plottingAdding procrustes support. Now all the output is registere to template skeleton.Bug fixes in CameraNetwork. Now calibration with arbitrary camera sequence is possible.Extras:If you are interested in the online annotation tool instead:DeepFly3DAnnotationTo see the dataset used in the paper:Dataverse