DOI: 10.5061/dryad.s09dr0q
title: Visual tracking of animals in three-dimensions using mobile handheld independent GoPro cameras and VSLAM software

description: I present the system PATMOS (Paths And Tessellated Meshes from ORB_SLAM2)
 for measuring three-dimensional paths of animals in situ using two
 handheld GoPro cameras and a small spatial reference
 object. Animal paths were triangulated from mobile camera
 positions obtained from a modified version of ORB_SLAM2, an open-source
 visual simultaneous localization and mapping software package. In
 addition to path calculation, this process provided a virtual
 three-dimensional surface approximation to the environment from which path
 to environment distances can be quantified. PATMOS can also fit a tranquil
 water’s surface to an analytic plane if there are a sufficient number of
 visible objects intersecting the water’s surface and can track objects
 over the water’s surfaces with a single camera by measuring the object
 with its reflection. This technology was highly portable, could follow
 moving animals, and gave comparable spatial and temporal resolutions to
 fixed camera systems that use commercial cameras. An investigation of
 falling objects yielded a gravitational constant measurement of 978±40
 cm/s2. I demonstrated PATMOS’s utility in terrestrial and aquatic
 environments by quantifying dragonfly flight characteristics and the
 inter-spatial distances between substrate and damselfish. 
description: The python executable
 for PATMOS is <b>PATMOS_MIDOM_200324.zip</b> This alone cannot run ORB_SLAM2 but it can visualize the data and allow the user to hand track the environment.  To load the virtual Linux environment that has ORB_SLAM2 and PATMOS already set up use this file. This is the Virtual Box Appliance file to load this Linux version on your computer.  <b>patmos.ova</b> The instructions on how to use these and install these files are in “Manual for PATMOS_MIDOM 200414.docx” You will then have to modify the ProjectMain.xlsx file as discussed in the Manual. To visualize the data using the PATMOS executable you need to put all these files from Dryad into the same folder that you will specify in ProjectMain.xlsx:   <b>Riparian environment:</b> Project name in ProjectMain.xlsx: tom_245_jill_189_v1 h5 files example_GoProJill_187_H.h5 example_GoProJill_189_H.h5 example_GoProTom_243_H.h5 example_GoProTom_245_H.h5 data_tom245_jill189.H5 map files: tomGP243_ltp0001rgb_topcut.stl tomGP243_ltp0001rgb_topcut.txt   <b>Marble dropping environment:</b> Project name in ProjectMain.xlsx: jill_320_tom_376 h5 files: data_jill_320_tom_376_2.h5 metadata_jill_320_HCO.h5 metadata_tom_376_HCO.h5 map files ML_ORB_jill_320_tom_376.txt ML_ORB_jill_320_tom_376.stl   And for running the ORB_SLAM2 in Virtual Box you will need to put these files from Dryad in the same video folder: Riparian environment: GoProJill/GOPR0187.MP4 GoProJill/GOPR0189.MP4 goprotom/GOPR0243.MP4 goprotom/GOPR0245.MP4   Marble dropping environment: GoProJill/GOPR0320.MP4 goprotom/GOPR0376.MP4   There should also be some preloaded map files for some of these projects in the Ubuntu file including this one for the marble drop experiment Map_ORB_jill_320_tom_376.bin   ProjectMain This is a spreadsheet that informs the analysis program which files to use.   <b>Figure 2</b> <b>Path data for blender visualization</b> Amberwing path data: For visualizing paths over meshes in blender. filenameq_CSV_MeshLab_rainbowandBlack.txt This consists of six paths that have been broken up into mirrored and regular counterparts. The blender file the paths have been rendered in: Amberwing5Vslam.blend The OpenMVS mesh file of the environment. scene_dense_mesh_textureML_AW.obj The STL was derived from KeyFrameTrajectory_tom_318_jill_261_190115_cleanup3.txt The waterplane was derived from KeyFrameTrajectory_tom_318_jill_261_190115_water_200522.txt The ORBSTL is KeyFrameTrajectory_tom_318_jill_261_190115_cleanup3.stl   Damselfish: The blender file is: DamselVsOCtRefined2.blend The OpenMVS mesh is: scene_dense_mesh_refine_texture_DF.obj   <b>Figure 3:</b> The data for this figure is in FiducialDistribution.xlsx Graphed in <i>Graphing_Fiducial_Distribution4BBpaper.py</i>   Distance between cameras:   ORB_SLAM2 measured data in it: Distance between Cameras.xlsx   The Physical measurements for the measurement rig: camera distances measured.xlsx   The graphs were made in <i>Graphing_Distance2STL_4Paper.py</i>   <b>Figure 4:</b> Marble dropping related figure:  Blender file: marbelsRefined_withGoProSTL4.blend   Marble data statistics: Statistics from the graph coming from <i>MarblesAngle_vs_g_and_errorDistance201021y.py</i>: General marble statistics: marble statisticsBBresub.xlsx   <b>Figure 5:</b> <b>The distance of paths to STL plots:</b> The graphs are made in <i>Graphing_Distance2STL_4Paper.py</i> <b>Amberwing:</b> Z-direction distance Amberwing13_INFO.xls Amberwing43_INFO.xls   <b>Damselfish:</b> Full Distance to STL. This is the OpenMVS STL distance without error. DamselPathsMVSn2_INFO.xls This is the PATMOS STL distance with error. DamselPathsn2_INFO.xls   <b>Figure 6:</b> Data are deposited in Sam_20377RC_Sky_20232RCFR_fh5_MapCovisibilityCharacteriztion_INFO.xls   <b>Sup Figure 1:</b> The map error graph is meant to give a complete idea of the map's error and reliability. This graph consists of: Map points in percentage levels Camera position colored with the same levels and joined together with a line The fiducial object with map points Color underlay of map errors The paths are put in there with the error with the same scaling as the background error estimate.   Amberwing: tom_319_jill_262_F_MapCharacteriztion_INFO.xls Marbles: jill_320_tom_376_2_MapCharacteriztion_INFO.xls Damselfish: Sam_20377RC_Sky_20232RCFR_fh5_MapCharacteriztion_INFO.xls   <b>Sup Figure 2:</b> <b>Difference between openMVS maps and PATMOS maps</b>   Damselfish: The map is: scene_dense_mesh_refine_texture_DF.obj   The map is compared with scene_dense_mesh_refine_Damselnew.stl The disparity information is in a numpy array saved in: Damseldisparitymap.npy and in dictionary form, listing a ; separated value with x;y;(distance between STL): DamseldisparitymapDict.npy Blender file: DamselMapComparison3_viridis.blend   <b>Marbles</b> The map is compared with the OpenMVS map: scene_dense_mesh_refine_Marbles5Rescale.stl The disparity information is in a numpy array saved in: marbledisparitymap.npy and in dictionary form, listing a ; separated value with x;y;(distance between STL): MarbledisparitymapDict.npy Blender file: marbelsMapComparisonnewcolor_newColor.blend   Amberwing: Amberdisparitymap.npy and in dictionary form, listing a ; separated value with x;y;(distance between STL): AmberdisparitymapDict.npy   The disparity stats calculation is done in Displarity_path_calculation.py Blender file: Amberwing7Vslam_viridis.blend   Other Statistical results not in figures:   Mean time and path length of falling marbles Found in the general info sheet in: marPathLength_INFO.xls   <b>Maps associated with these projects</b> Amberwing: The map for the amber wing data was not very good.   It did not refine in OpenMVS, It did not include the water surface where the amberwings were found flying, and it did not give good results of the branches in the water.   The map used for blender was: scene_dense_mesh_textureML_AW.obj other versions: scene_dense_mesh_Amberwing3.stl and was put in the Red folder Maps and rescaled to: scene_dense_mesh_Amberwing3Rescale.stl The stl was derived from KeyFrameTrajectory_tom_318_jill_261_190115_cleanup3.txt The water plane was derived from KeyFrameTrajectory_tom_318_jill_261_190115_water_200522.txt   Damselfish: OpenMVS file is: scene_dense_mesh_refine_Damsel.stl and is rescaled as: scene_dense_mesh_refine_Damselnew.stl The PATMOS STL was derived from KFT_ORB_Sam_20377RC_Sky_20232RC_cut.txt That has various parts of the point cloud cut. The PATMOS SLT with error are: KFT_ORB_Sam_20377RC_Sky_20232RC_cut.stl KFT_ORB_Sam_20377RC_Sky_20232RC_cut_err.npy   Marbles: The OpenMVS mesh is scene_dense_mesh_refine_texture_Mar.obj scene_dense_mesh_refine_Marbles5.stl rescaled scene_dense_mesh_refine_Marbles5Rescale.stl   For the ORB_STL: Use as an input ML_ORB_jill_320_tom_376_2.txt: The PATMOS SLT with error are:   ML_ORB_jill_320_tom_376_2.stl ML_ORB_jill_320_tom_376_2_err.npy readmeVSLAMtracking readme for the entire submission