Package: pytheia
pyTheia - A Python Structure-from-Motion and Geometric Vision Swiss KnifepyTheia is based onTheiaSfM.
It contains Python bindings for most of the functionalities of TheiaSfM and more.The library is still in active development and the interfaces are not yet all fixedWith pyTheia you have access to a variety of different camera models, structure-from-motion pipelines and geometric vision algorithms.Differences to the original library TheiaSfMpyTheia does not aim at being an end-to-end SfM library. For example, building robust feature detection and matching pipelines is usually application and data specific (e.g. image resolution, runtime, pose priors, invariances, ...). This includes image pre- and postprocessing.pyTheia is rather a "swiss knife" for quickly prototyping SfM related reconstruction applications without sacrificing perfomance.
For example SOTA feature detection & matching, place recognition algorithms are based on deep learning, and easily usable from Python. However, using these algorithms from a C++ library is not always straighforward and especially quick testing and prototyping is cumbersome.What was removedHence, we removed some libaries from the original TheiaSfM:SuiteSparse: Optional for ceres, however all GPL related code was removed from src/math/matrix/sparse_cholesky_llt.cc (cholmod -> Eigen::SimplicialLDLT). This will probably be slower on large problems and potentially numerically a bit more unstable.OpenImageIO: was used for image in and output and for recitification.RapidJSON: Camera intrinsic in and output. Is part of cereal headers anyways.RocksDB: Used for saving and loading extracted features efficiently.Changes to the original TheiaSfM libraryGlobal SfM algorithms:LiGT position solverLagrange Dual rotation estimatorHybrid rotation estimatorPossibility to fix multiple views in Robust_L1L2 solverNonlinear translation solver can fix multiple view or estimate all remaining views in reconstructionCamera modelsDouble SphereExtended UnifiedOrthographicBundle adjustmentUsing a homogeneous representation for scene pointsExtracting covariance informationPossibility to add a depth prior to 3D pointsPosition prior for camera poses (e.g. for GPS or known positions)GeneralAdded timestamp, position_prior_, position_prior_sqrt_information_ variables toViewclass
Eigen::Matrix3d position_prior_sqrt_information_;Added inverse_depth_, reference_descriptor, reference_bearing_ variables toTrackclassAdded covariance_, depth_prior_, depth_prior_variance_ toFeatureclassAbsolute Pose solversSQPnPUncalibratedPlanarOrthographic PoseUsage ExamplesFull reconstruction example: Global, Hybrid or Incremental SfM using OpenCV feature detection and matchingHave a look at the short example:sfm_pipeline.py.
Download the south_building dataset fromhere.
Extract it somewhere and run:pythonpytest/sfm_pipeline.py--image_path/path/to/south-building/images/Creating a cameraThe following example show you how to create a camera in pyTheia.
You can construct it from a pt.sfm.CameraIntrinsicsPrior() or set all parameters using respective functions from pt.sfm.Camera() class.importpytheiaasptprior=pt.sfm.CameraIntrinsicsPrior()prior.focal_length.value=[1000.]prior.aspect_ratio.value=[1.]prior.principal_point.value=[500.,500.]prior.radial_distortion.value=[0.,0.,0.,0]prior.tangential_distortion.value=[0.,0.]prior.skew.value=[0]prior.camera_intrinsics_model_type='PINHOLE'#'PINHOLE', 'DOUBLE_SPHERE', 'EXTENDED_UNIFIED', 'FISHEYE', 'FOV', 'DIVISION_UNDISTORTION'camera=pt.sfm.Camera()camera.SetFromCameraIntrinsicsPriors(prior)# the camera object also carries extrinsics informationcamera.SetPosition([0,0,-2])camera.SetOrientationFromAngleAxis([0,0,0.1])# project with intrinsics image to camera coordinatescamera_intrinsics=camera.CameraIntrinsics()pt2=[100.,100.]pt3=camera_intrinsics.ImageToCameraCoordinates(pt2)pt2=camera_intrinsics.CameraToImageCoordinates(pt3)# project with camera extrinsicspt3_h=[1,1,2,1]# homogeneous 3d pointdepth,pt2=camera.ProjectPoint(pt3_h)# get a ray from camera to 3d point in the world frameray=camera.PixelToUnitDepthRay(pt2)pt3_h_=ray*depth+camera.GetPosition()# == pt3_h[:3]Solve for absolute or relative camera posepyTheia integrates a lot of performant geometric vision algorithms.
Have a look at thetestsimportpytheiaaspt# absolute posepose=pt.sfm.PoseFromThreePoints(pts2D,pts3D)# Kneippose=pt.sfm.FourPointsPoseFocalLengthRadialDistortion(pts2D,pts3D)pose=pt.sfm.FourPointPoseAndFocalLength(pts2D,pts3D)pose=pt.sfm.DlsPnp(pts2D,pts3D)...andmore# relative posepose=pt.sfm.NormalizedEightPointFundamentalMatrix(pts2D,pts2D)pose=pt.sfm.FourPointHomography(pts2D,pts2D)pose=pt.sfm.FivePointRelativePose(pts2D,pts2D)pose=pt.sfm.SevenPointFundamentalMatrix(pts2D,pts2D)...andmore# ransac estimationparams=pt.solvers.RansacParameters()params.error_thresh=0.1params.max_iterations=100params.failure_probability=0.01# absolute pose ransaccorrespondences2D3D=pt.matching.FeatureCorrespondence2D3D(pt.sfm.Feature(point1),pt.sfm.Feature(point2))pnp_type=pt.sfm.PnPType.DLS#  pt.sfm.PnPType.SQPnP,  pt.sfm.PnPType.KNEIPsuccess,abs_ori,summary=pt.sfm.EstimateCalibratedAbsolutePose(params,pt.sfm.RansacType(0),pnp_type,correspondences2D3D)success,abs_ori,summary=pt.sfm.EstimateAbsolutePoseWithKnownOrientation(params,pt.sfm.RansacType(0),correspondences2D3D)...andmore# relative pose ransaccorrespondences2D2D=pt.matching.FeatureCorrespondence(pt.sfm.Feature(point1),pt.sfm.Feature(point2))success,rel_ori,summary=pt.sfm.EstimateRelativePose(params,pt.sfm.RansacType(0),correspondences2D2D)success,rad_homog,summary=pt.sfm.EstimateRadialHomographyMatrix(params,pt.sfm.RansacType(0),correspondences2D2D)success,rad_homog,summary=pt.sfm.EstimateFundamentalMatrix(params,pt.sfm.RansacType(0),correspondences2D2D)...andmoreBundle Adjustment of views or pointsimportpytheiaasptrecon=pt.sfm.Reconstruction()# add some views and pointsveiw_id=recon.AddView()...track_id=recon.AddTrack()...covariance=np.eye(2)*0.5**2point=[200,200]recon.AddObservation(track_id,view_id,pt.sfm.Feature(point,covariance))# robust BAopts=pt.sfm.BundleAdjustmentOptions()opts.robust_loss_width=1.345opts.loss_function_type=pt.sfm.LossFunctionType.HUBERres=BundleAdjustReconstruction(opts,recon)res=BundleAdjustPartialReconstruction(opts,{view_ids},{track_ids},recon)res=BundleAdjustPartialViewsConstant(opts,{var_view_ids},{const_view_ids},recon)# optimize absolute pose on normalized 2D 3D correspondencesres=pt.sfm.OptimizeAbsolutePoseOnNormFeatures([pt.sfm.FeatureCorrespondence2D3D],R_init,p_init,opts)# bundle camera adjust pose onlyres=BundleAdjustView(recon,opts,view_id)res=BundleAdjustViewWithCov(recon,view_id)res=BundleAdjustViewsWithCov(recon,opts,[view_id1,view_id2])# optimize structure onlyres=BundleAdjustTrack(recon,opts,trackid)res=BundleAdjustTrackWithCov(recon,opts,[view_id1,view_id2])res=BundleAdjustTracksWithCov(recon,opts,[view_id1,trackid])# two view optimizationres=BundleAdjustTwoViewsAngular(recon,[pt.sfm.FeatureCorrespondence],pt.sfm.TwoViewInfo())BuildingThis section describes how to build on Ubuntu locally or on WSL2 both with sudo rights.
The basic dependency is:http://ceres-solver.org/Installing the ceres-solver will also install the neccessary dependencies for pyTheia:gflagsglogEigensudoaptinstallcmakebuild-essential# cd to your favourite library foldermkdirLIBScdLIBS# eigengitclonehttps://gitlab.com/libeigen/eigencdeigen&&gitcheckout3.4.0
mkdir-pbuild&&cdbuild&&cmake..&&sudomakeinstall# libgflags libglog libatlas-base-devsudoaptinstalllibgflags-devlibgoogle-glog-devlibatlas-base-dev# ceres solvercdLIBS
gitclonehttps://ceres-solver.googlesource.com/ceres-solvercdceres-solver&&gitcheckout2.1.0&&mkdirbuild&&cdbuild
cmake..-DBUILD_TESTING=OFF-DBUILD_EXAMPLES=OFF-DBUILD_BENCHMARKS=OFF
make-j&&makeinstallLocal build without sudoTo build it locally it is best to set the EXPORT_BUILD_DIR flag for the ceres-solver.
You will still needsudo apt install libgflags-dev libgoogle-glog-dev libatlas-base-dev.
So go ask your admin ;)# cd to your favourite library folder. The local installation will be all relative to this path!mkdir/home/LIBScd/home/LIBS# eigengitclonehttps://gitlab.com/libeigen/eigencdeigen&&gitcheckout3.4.0
mkdir-pbuild&&cdbuild&&cmake..-DCMAKE_INSTALL_PREFIX=/home/LIBS/eigen/build&&make-jinstallcd/home/LIBS
gitclonehttps://ceres-solver.googlesource.com/ceres-solvercdceres-solver&&gitcheckout2.1.0&&mkdirbuild&&cdbuild
cmake..-DBUILD_TESTING=OFF-DBUILD_EXAMPLES=OFF-DBUILD_BENCHMARKS=OFF-DEXPORT_BUILD_DIR=ON
make-j# cd to the pyTheiaSfM foldercdpyTheiaSfM&&mkdirbuild&&cdbuildcmake-DEigen3_DIR=/home/LIBS/eigen/build/share/eigen3/cmake/..make-jHow to build Python wheelsLocal build with sudo installed ceres-solver and EigenTested on Ubuntu. In your Python >= 3.6 environment of choice run:shbuild_and_install.shIf you have problems like/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not foundon Ubuntu 22.04 in an Anaconda environment try:condainstall-cconda-forgelibstdcxx-ngWith DockerThe docker build will actually build manylinux wheels for Linux (Python 3.6-3.12).
There are two ways to do that. One will clutter the source directory, but you will have the wheel file directly available (./wheelhouse/).
Another drawback of this approach is that the files will have been created with docker sudo rights and are diffcult to delete:# e.g. for python 3.9dockerrun--rm-ePYTHON_VERSION="cp39-cp39"-v`pwd`:/homeurbste/pytheia_base:1.2.0/home/pypackage/build-wheel-linux.shThe other one is cleaner but you will have to copy the wheels out of the docker container afterwards:dockerbuild-tpytheia:0.1.
dockerrun-itpytheia:0.1Then all the wheels will be inside the container in the folder /home/wheelhouse.
Open a second terminal and rundockerps# this will give you a list of running containers to find the correct CONTAINER_IDdockercpCONTAINER_ID:/home/wheelhouse/path/to/result/folder/pytheia_wheels