Package: caliscope
January 28, 2024: If you are reading this, then I am actively (likeright now) in the process of switching some things over with PyPI and links, so names and references may be wonky. The package has now been renamedcaliscope(previouslypyxy3d). I've recently completed some major refactors that enable a pre-recorded workflow, split out the sychronized webcam recording into its own project (multiwebcam), and have started the ball rolling on a submission for theJournal of Open Source Software. I'd been wanting to change the name for awhile, and with all this transition it seems like the right time.CaliscopeMulticamera Calibration + Pose Estimation --> Open Source Motion CaptureAboutCaliscopeis a GUI-based multicamera calibration package that integrates with 2D landmark tracking tools to produce low-cost, open-source motion capture.Please see ourdocsfor details about installation, project setup, and general workflow.Demo Outputhttps://github.com/mprib/caliscope/assets/31831778/803a4ce8-4012-4da1-87b9-66c5e6b31c59The above was created using Caliscope, a 7 year old t440p laptop, and 4 webcams (~$25 each). This includes camera calibration, recording of synchronized frames (720p @ 24 fps), landmark detection, and point triangulation. Note that the webcam recording functionality is not in the current version, though will be restored in the future. Animated rig creation was done using an early stage Blender add-on project calledRigmaroleQuick StartPlease note that given the size of some core dependencies (OpenCV, Mediapipe, and PySide6 are among them) installation and initial launch can take a while.Basic Installation and LaunchWindows# Open Command Prompt and navigate to directory that will hold venv# this does not need to be the same as where your project workspace is heldcdpath\to\your\project# Create a virtual environment named 'env' using Python 3.10"C:\Path\To\Python3.10\python.exe"-mvenv.venv# Activate the virtual environment.\env\Scripts\activate# Your virtual environment is now active.# You can install using pippipinstallcaliscope# Launch from the command linecaliscopeMacOS/Linux# Open Command Prompt and navigate to directory that will hold venv# this does not need to be the same as where your project workspace is heldcdpath/to/your/project# Create a virtual environment named 'venv' using Python 3.10/path/to/python3.10-mvenv.venv# Activate the virtual environmentsource.venv/bin/activate# Your virtual environment is now active.# You can install using pippip3installcaliscope# Launch from the command linecaliscopeBasic StepsOnce the GUI launches, navigate to File->New/Open Project and create a folder to hold your projectA basicproject structurewill be created hereDefine a Charuco calibration board via the Charuco tab and print it out, fixing it to something flatRecord footage for the calibration according to the guidelines for theintrinsicandextrinsiccalibrations.Record synchronized motion capture trialsA companion project (multiwebcam) has been set up to facilitate this though is still in early stagesStore video files within the project folder and reload the workspaceRun autocalibration on all cameras within the Cameras tabRun "Calibration Capture Volume" from the Workspace TabSet origin within the capture volume (optional but helpful)Run post-processing on individual recordings to generate 3D trajectory outputKey FeaturesCalibration board creationEasy creation ofpngfiles for ChArUco calibration boardsboard definition can be changed across intrinsic and extrinsic calibration allowing greater flexibiltiyIntrinsic Camera CalibrationAutomated calculation of camera intrinsic properties from input videoOptical CenterFocal LengthLens DistortionVisualization of distortion model to ensure reasonablenessExtrinsic Camera CalibrationAutomated bundle adjustment to estimate 6 DoF relative position of camerasVisualizer to inspect the estimates from the bundle adjustmentSetting of the World Origin within the visualizer to simplify data processing3D TrackingTracker API for integrating alternate tracking methods3 sample implementations with Google Mediapipe (Hands/Pose/Holistic)Automated application of landmark tracking to synchronized videosTriangulation of 3D landmark position based on calibrated camerasGap-filling and butterworth filtering to smooth trajectory estimatesTrajectory Outputoutput to.trcfile format for use in biomechanical modellingoutput to tidy.csvformat with well-labelled headers for straightforward integration with other workflowscompanion projectRigmarolein development to facilitate creation of animated rigs in BlenderLimitationsRequires Frame SyncThe workflow currently requires you to provide your own synchronized frames or to providea filethat specifies the time at which each frame was read so that caliscope can perform the synchronization itself. There are plans to manage this synchronization automatically through audio files, though that has not yet been implemented.Currently only using MediapipeGoogle's Mediapipe provides a relatively easy and efficient method for human subject tracking, though for many uses it is limiting. Caliscope has a general Tracker base class that is implemented in a few versions (Pose/Hands/Holistic). This has provided a proof of concept implementation of markerless tracking, though for more robust use the roadmap calls for integration with more powerful tools such asMMPoseandDeepLabCut.Reporting Issues and Requesting FeaturesTo report a bug or request a feature, pleaseopen an issue. Please keep in mind that this is an open-source project supported by volunteer effort, so your patience is appreciated.General Questions and ConversationPost any questions in theDiscussionssection of the repo.AcknowledgmentsThis project was inspired byFreeMoCap(FMC), which is spearheaded byJon Matthis, PhDof the HuMoN Research Lab. The FMC calibration and triangulation system is built uponAnipose, created by Lili Karushchek, PhD. Caliscope was originally envisioned as an alternative calibration tool to Anipose that would allow more granular estimation of intrinsics as well as visual feedback during the calibration process. Several lines of of the original Anipose triangulation code are used in this code base, though otherwise it was written from the ground up. I'm grateful to Dr. Matthis for his time developing FreeMoCap, discussing it with me, pointing out important code considerations, and providing a great deal of information regarding open-source project management.LicenseCaliscope is licensed under the permissiveBSD 2-Clause license. The triangulation function was adapted from theAniposecode base which is also licensed under the BSD-2 Clause. A primary dependency of this project is PySide6 which provides the GUI front end. PySide6 is licensed under theLGPLv3. Caliscope does not modify the underlying source code of PySide6 which is available viaPyPI.