TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICSMARIAN BITTNER 1,2,3, *,† , WEI-TSE YANG 2,† , XUCONG ZHANG 2 , AJAY SETH 3 ,JAN VAN GEMERT 2 , AND FRANS C. T. VAN DER HELM 3

Abstract. Markerless estimation of 3D Kinematics has the great potential
to clinically diagnose and monitor movement disorders without referrals to
expensive motion capture labs; however, current approaches are limited by
performing multiple de-coupled steps to estimate the kinematics of a personfrom videos. Most current techniques work in a multi-step approach by firstdetecting the pose of the body and then fitting a musculoskeletal model tothe data for accurate kinematic estimation. Errors in training data of thepose detection algorithms, model scaling, as well the requirement of multiple
cameras limit the use of these techniques in a clinical setting. Our goal is
to pave the way toward fast, easily applicable and accurate 3D kinematic
estimation . To this end, we propose a novel approach for direct 3D humankinematic estimation D3KE from videos using deep neural networks. Our
experiments demonstrate that the proposed end-to-end training is robust andoutperforms 2D and 3D markerless motion capture based kinematic estimation
pipelines in terms of joint angles error by a large margin (35% from 5.44 to
3.54 degrees). We show that D3KE is superior to the multi-step approach andcan run at video framerate speeds. This technology shows the potential forclinical analysis from mobile devices in the future.

1. Introduction
3D Human kinematics involves measuring joint angles between body segments,
which is essential in the day-to-day practice of experts. Skilled physicians could
judge, just by looking at a specific motion of their patient, whether it is healthy
or abnormal. Skilled sports coaches can help their coachees achieve better performance and lower injury risk by evaluating their movements through observation.
However, these visual examinations of human kinematics remain inherently subjective, leading to variation between and within human observers. Modern systems
and sensors could reduce these variations through more objective observations. Yet,these systems make the measurement of human motion more costly and more timeconsuming. A system with the availability and ease of use of visual estimation
would help physicians and coaches make more objective observations more often,
ultimately raising their own and their subjects quality of life. Digital cameras
have made the estimation of human kinematics more accessible but come at the1
Vicarious Perception Technologies (VicarVision), 1015 AH Amsterdam, The
Netherlands
2Computer Vision Lab, Delft University of Technology, 2628 XE Delft, The
Netherlands
3
Biomechanical Engineering, Delft University of Technology, 2628 CN Delft, The
Netherlands*Corresponding author: mbittner.work@gmail.com†These authors contributed equally to this work.

1

2

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

cost of reduced accuracy. Compared to the more traditional Optical Motion capture (OMC) systems, markerless motion capture (MMC) systems do not require
specialized cameras and markers attached to the subject being monitored, but use
normal RGB cameras in combination with image-based automatic pose estimation
algorithms. Instead of specific markers, pose estimation algorithms detect the centers of major joints of the human body, such as the shoulders, hips, and knees.
These detected centers are usually referred to as key points.
Multiple commonly used markerless motion capture methods rely on 2D pose
estimation methods [27, 45, 46, 26]. Often these methods still need more than one
camera to generate a good estimation of the keypoints in 3D, which again requires
additional cameras to be set up. On the other hand, an increasing number of methods are using single-view (monocular) 3D pose estimation methods [21, 32, 43],
which allow to estimate a 3D pose just by using a single camera. This makes MMC
systems faster and more accessible as they do not require the additional time andexpertise to place markers on the subject or calibrate multiple cameras. However,
MMC systems assume that current pose estimation algorithms can accurately replace markerless motion capture systems for, e.g., biomechanical applications [52,13, 60].
Commonly used pose estimation algorithms introduce mistakes in kinematic estimation pipelines due to systematic errors in their predictions. To detect key
points, most pose estimation methods are trained on a combination of images of aperson and ground truth annotations which map pixels in the image to their corresponding joint center.These ground truth annotations are often manually conductedby non-expert annotators, leading to errors caused by personal biases for training
and inaccuracies in the pose estimations [13]. For example, Needham et al. [41]compared three often used pose estimation algorithms OpenPose [10], DeepLabCut [38] and AlphaPose [17] algorithm against an OMC system and showed errors
in the estimation of joint centers of 30 mm to 50 mm with variations in 12 mm
to 25 mm in marker placement. Cronin [13] provides an overview of additional
problems with 2D pose estimation for kinematic analysis. These differences aremost likely due to a difference between the application that pose estimation algorithms are often developed for and their application to, e.g., the biomedical domain, which has different accuracy requirements [52]. Wade et al. [60] proposed
to solve this problem by re-annotating existing large-scale datasets, this, however,is a time-consuming process, when for example considering the COCO-keypoint
dataset https://cocodataset.org/#keypoints-2020 (accessed on 2 December
2022 ) consists of more than 250.000 labeled poses. For the evaluation of pose estimation algorithms, these labeling errors will just appear as a baseline error that
all algorithms training on the same data will have. However, for applications in thebiomedical domain and in situations such as kinematic estimation, where the poseis just an intermediate step errors can propagate to subsequent tasks.
Errors in the estimated pose cannot not be corrected by most kinematic estimation pipelines because they all roughly follow a ‘multi-step’ approach. The‘multi-step’ approach consists of• Detection of the 3D pose (in one or more steps);• (Optional) modeling of the pose with a (musculo)skeletal model.
• Calculation of kinematics and/or downstream tasks such as gait parameters
or dynamics.

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

3

For example, Kidzinski et al. [27] used OpenPose to first predict key points froma video and then trained a convolutional neural network (CNN) to predict the walking parameters of patients with cerebral palsy. Liao et al. [32] first model the 2D
pose in OpenPose then create a 3D pose using data-driven matching and finallyestimate 3D gait parameters. Noteboom et al. [43] first used VideoPose3D [49] toestimate a 3D pose, followed by modeling in OpenSim [54] for the estimation of dynamics from a single camera. Pagnon et al. [45, 46] developed the handy Pose2Sim
tool, which first combines 2D OpenPose pose estimations from multiple cameras
into a 3D pose then models it in OpenSim. Because the pose estimation step isde-coupled from kinematic estimation, errors in pose estimation propagate throughto the estimation of kinematics. Uchida and Seth [57] showed that 20 mm of marker
uncertainty leads to a variation of 15.9◦ in peak ankle plantarflexion angle and impacts downstream tasks such as joint moment estimation. Della Croce et al. [14]showed precision variation 13 mm to 25 mm, which leads to differences in estimatedjoint angles up to 10◦ . Fonseca et al. [18] showed that misplacement of markers up
to 10 mm can lead to errors of 7◦ depending on the marker. With estimation errorsof 30 mm to 50 mm in keypoint estimation [41], it is to be expected that these errors
will substantially influence kinematic estimation from markerless motion capture.
Low-pass filter [40, 45] or bi-directional Kalman-filter [40] has been applied to compensate for noisy key point estimations, but cannot correct for faults in keypoint
detection. Subsequent modeling and kinematic calculation steps can only compensate for these inaccuracies. This ‘multi-step’ approach is probably inspired by thesteps of a traditional OMC method, as in the traditional OMC systems the pose
detection step is done using a different system and is thus isolated from the othersteps. In camera-based kinematic estimation pipelines, however, the de-coupling ofindividual steps is no longer necessary.
Deep neural networks have often demonstrated their ability to outperform multistep systems, by implicitly learning individual steps through end-to-end training
between an input and the desired output [55, 29]. The main strength of deep neural networks lies in their ability to break down a highly complex task, in this case,the estimation of kinematics from videos, into a sequence of simpler tasks, without the need for intermediate ‘hand-crafted’ representations [30, 2]. Due to the fully
differentiable nature of neural networks, it means that an error in estimation during training can influence all stages of the network and adjust them accordingly [2].
This allows deep neural networks to directly estimate kinematics.
In this work, we challenge the notion of the classical multi-step approach of pose
estimation, fitting of a musculoskeletal model and kinematic estimation. To thisend, we propose a novel end-to-end method that allows for direct estimation of human kinematics, which is directly optimized for kinematic estimation while treatingpose estimation only as an auxiliary task to constrain the estimations of the network. Figure 1 shows a general overview of our method.

4

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

Figure 1. Overview of the proposed direct 3D human kinematics estimation (D3KE). Instead of using the common ’multi-step’
approach of predicting pose, fitting it to a model, and estimating
kinematics, our D3KE directly estimates the kinematics. Errors inearlier steps of the multi-step approach propagate to later steps;
in contrast, our method can correct for errors occurring anywhere
between input and output.

Contributions. To the best of our knowledge, we are the first to present an endto-end trainable network that directly generates joint angles, joint positions, scale
factors and marker positions of a biomechanical model from a monocular video.
We propose a method that directly regresses from a video to joint angles andscales using deep neural networks. We investigate the influence of various temporalsmoothing methods to increase the accuracy of our algorithm. We introduce a novel
type of network layer that allows for the calculation of the 3D pose from estimatedkinematics during the training process to train the network simultaneously on thepose and kinematic labels.

2. Materials and Methods
Our method takes videos from a single camera as input and directly estimates
joint angles, which we call direct 3D kinematic estimation (D3KE). The proposed
method first coarsely estimates kinematics per frame by using a convolutional neuralnetwork, and then it uses a sequence network with temporal relations across frames
to re-fine kinematic estimations at each frame. An overview of our method is shownin Figure 2. Both networks estimate the scale of body segments, joint angles, and arotation matrix from the pelvis to the ground, those serve as input for a skeletalmodel layer in both networks that allows for additional supervision on the pose of
a subject.

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

5

Figure 2. Taking a single view video as input, D3KE consists of
one convolutional neural network and one sequential network. Per
frame, D3KE outputs joint angles and scales of individual bones ina skeletal model(scale factors) with a convolutional network. Additionally, joint angle and scale factor are converted to a pose throughthe skeletal-model kinematics (SM) layer. A series of frame estimations in time are then fed into a sequential network to smooththe estimations and reduce artifacts if one limb occludes anotherin the view of the camera (self-occlusion).

In this section, we first describe the deep learning architecture, including a detailed description of the skeletal-model layer. We then describe how the groundtruth data was generated and which pre-processing and hyperparameters wereused for training. Lastly, we describe the dataset used for training and testing
our method.
2.1. Network Structure. Convolutional neural networks(CNNs) have shown good
accuracy for 2D and 3D pose estimations [10, 51, 39, 42, 48] from single input images. Conventionally 2D CNNs are used for pose estimation tasks, that takes asingle image as an input and predict the pose of one or multiple people in theimage. For our method, we choose a per-frame convolutional network to coarsely
predict the joint angle and scaling parameters. Inspired by [51], we choose a standard pre-trained ResNeXt-50 [62] as our convolutional backbone.
To fine-tune the per-frame predicted joint angles and scaling parameters we add
a sequential network. Sequential networks are used in pose estimation to ‘lift’ anestimated 2D pose to 3D [11, 12]. Recent research combines temporal information
with lifting to improve accuracy during frames where one limb occludes anotherin the view of the camera (self-occlusion) or where not all key points were detected [31, 33, 49]. In contrast to CNNs, these sequential networks do not take asingle frame as input but exploit temporal dependencies in the data for their prediction. As the convolutional network outputs per-frame estimates, it cannot taketemporal information into account. We add a sequential network to our architecture
to refine a sequence of estimations made by the convolutional model. Inspired byworks on temporal lifting we experimentally evaluate three sequential networks; an

6

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

LSTM [23], a Temporal Convolutional Network (TCN) [49] and a Transformer [31]to refine the predicted joint angles and scale factors.
Both the convolutional and the sequential network contain a specialized layerthat allows each network to perform the kinematic transformations of a musculoskeletal model. Therefore, at train time both networks can be supervised not only
on the estimated joint angles but also on a resulting pose.
Both convolutional and sequential networks are supervised by losses of joint
positions, marker positions, body scales and joint angles. The overall objectivefunction L can be expressed in the equation(1)

L = λ1 Ljoint + λ2 Lmarker + λ3 Lbody + λ4 Langle ,where λ1 , λ2 , λ3 , λ4 are weights of losses. We use the root-relative L1 loss in Equation (2) to define the loss of marker position Lmarker and the loss of joint position
Ljoint . The estimations ŷ and the labels y are first subtracted with each root position ŷroot , yroot . For the loss of body scales Lbody and joint angles Langle , wecalculate the L1 norm.

(2)l = ∥(ŷ − ŷroot ) − (y − yroot )∥1

The objective of a neural network during training is to minimize the loss function;in our case, the difference between estimated and ground truth joint angles. However, this joint angle loss cannot capture the underlying relations and constraints
of individual angles, dictated by the human musculoskeletal system. Intuitively,
small changes in the angles of spine, shoulder, and elbow can accumulate and leadto large differences in the position of the hand, as illustrated in Figure 3. To address this issue, we propose to use a skeletal-model layer to perform the kinematictransform of a musculoskeletal model.

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

7

Figure 3. Our skeletal-model layer uses an internal representation of a skeletal model to convert the predicted joint angles andscale factors to the positions of individual markers on segments ofthe skeletal model. This allows our method to be supervised during
training not only on errors(losses) in the estimation of joint angles
but also on errors in the resulting pose. On the right, we show theadditional error that is created between estimations (gray) andground truth (blue). This auxiliary estimation of the pose as 3D
marker positions helps to constrain the estimation of joint angles
as small changes in proximal joints can have a large effect on amarker at more distal positions.

Skeletal-Model Layer. The skeletal-model layer allows us to convert predicted joint
angles into marker positions on a skeletal model and add them as an additionalloss term. This loss term represents the cumulative effect of small joint angle
changes on the final pose, indirectly imposing the constraints of a skeletal-modelon the predictions of the network. As the skeletal-model layer does not contain any
learnable parameters, i.e., it cannot change during network training. The accuracyof the predicted pose is completely determined by the input to the skeletal-model
layer; thus, the pose prediction is only an auxiliary task.
A skeletal model consists of body segments, motions between different bodysegments (joints) and points with a vector from a center of its anchor body segment(markers). Given body scales β, joint angles θ and rotation matrix Rground←pelvis ,we use the skeletal-model layer to calculate marker positions and joint positions.
In the following variables with a hat (x̂) denote estimated values, variables without
the hat (x) denote the predefined variable from the musculoskeletal model.

8

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICSFirst, the translation part T in the transformation from the joint to the bodydepends on the subject’s body scale. For example if the subject has longer legs,
the center of the femur will be farther from the hip joint. We can update thetranslation part T̂ by comparing the ratio between predicted body scales β̂ anddefault body scales β in Equation (3), where ⊙ is elementwise multiplication, and ⊘is elementwise division.
(3)T̂ = T ⊙ (β̂ ⊘ β)Then, we create a matrix to represent spatial transformation of motions Rm otion
using Equation (4) A1 , A2 , A3 are the predefined axes θ̂1 , θ̂2 , θ̂3 and predicted angels
per degree of freedom per joint in axis-angle notation. G(A, θ) is the standard
function converting an axis-angle representation to a 3x3 transformation matrix.
(4)

 R3 R2 R1
Rmotion = 

0

(5)

R1 = G(A1 , θ1 )

(6)

R2 = G(R1 A2 , θ2 )(7)

R3 = G(R2 R1 A3 , θ3 )

0


1 4×4

Then, we can calculate the estimated transformation from the body to its parent
body R̂parent←child in Equation (9) using Equation (8) with Oparent , Ochild denoting
predefined orientations from and T̂parent , T̂child the predicted translations from thejoint to the parent/child. F (Oa ) the conversion from euler angles to a 3 × 3
Rotation matrix.
(8)
Rparent/child←joint (Oparent/child , Tparent/child ) = F (Oparent/child )0(9)

Tparent/child 


1
4×4

−1
Rparent←child = Rparent←joint Rmotion Rchild←joint

We measure the spatial transform by traversing from the root (pelvis) to leaf
nodes (hands and feet) in the level order. In D3KE, we directly infer the rotation
matrix from the pelvis to the ground Rground←pelvis . Rground←pelvis can initiallybe expressed in Equation (10), where I denotes the identity matrix. The rotation
part of Rchild←joint is also a 3 × 3 identity matrix in our musculoskeletal model.
Our method aims to predict the root-relative position so the translation part canbe ignored during the prediction. Moreover, in our musculoskeletal model, only
joint angles of the pelvis are unbounded in [−∞,∞]. Predicting three unbounded
angles to form the rotation matrix in Equation (4) will have the problem of discontinuity [65]. Thus, we directly predict the rotation matrix Rground←pelvis .
(10)−1
Rground←pelvis = I4×4 Rmotion Rpelvis←jointTOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

9

Last, a marker with a vector of d⃗ from the center of the body is also dependent on
the body scales. The predicted vector of dˆ is updated in Equation (11). The position
ˆof the predicted point is calculated in Equation (12) with R̂parent←child and d.
dˆ = d⃗ ⊙ (β̂ ⊘ β)

(11)

 
(12)

Y

P =

parent,child∈path

2.2. Network Training.

d⃗Rparent←child 
 
1 4x1

2.2.1. Ground Truth Generation. For training our method, we need to create custom ground truth data that contains all outcomes that our network is predicting
since they are not available in publicly available datasets. Most pose estimation
datasets, provide only video and marker positions from optical motion capture(OMC) system. For training our method, we need the joint angle and the scalesof individual bones, a rotation matrix of the pelvis to the ground as well as themarker positions corresponding to them. To generate these, we model the OMC
data, represented as a 3D human mesh model in the OpenSim software [54] anduse inverse kinematics [36, 1] to generate joint angles. The following describes eachstep in more detail.
First, we create a general (musculo)skeletal model to fit the data using the
OpenSim software [54, 16]. As we are interested in capturing the complete motionof the human subject, we model the full body. With the OpenSim software [16, 54]
we create a full-body musculoskeletal model (MSM) by merging existing models of
upper limbs and lower limbs [3, 4, 15, 24, 63] and thoracolumbar spine [8, 7, 9].
We add wrist and hand [20, 44] models to the MSM, which are not used for groundtruth generation, for the sake of aesthetics. The full-body model contains all bones
in a skeletal system from the head to feet and from the upper arms to the hands.
We do not model every degree of freedom between vertebrae to avoid expensive
computation and the requirement of at least three markers to measure the motionsof one vertebra. Instead, we separate the spine from the fifth lumbar to the firstcervical vertebra into nine segments.
Then, we fit our data to the musculoskeletal model. Instead of using the OMC
marker data directly, we use OMC marker converted to 3D human mesh representations using the MoSh++ [34] method, to make scaling the model to individual
participants more time efficient and allow us to define an arbitrary number of virtual markers. We fit our data to the musculoskeletal model, by first defining virtual
markers on the vertices of the 3D mesh representation. We then used these virtual
markers as input for the OpenSim software. Then, we used the OpenSim internal
scaling tool to scale the proportion of individual body segments according to the
distances of virtual markers on the 3D mesh. As the sizes of individual body parts
vary across individuals, this step must be conducted individually for each subject
in the dataset. We define the ratio in dimensions between the default and scaled
body segments as scaling factors. Finally, we used the inverse kinematics solver forthe calculation of joint angles. During this process, the MSM is moved for eachtime step to a position that minimizes the sum of weighted squared errors between
the virtual markers on the 3D mesh and markers defined on the musculoskeletal

10

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

model. All joint angles where segments had a higher squared error than 2 cm weredisregarded in the analysis.
The final ground truth values were the calculated joint angles, the scaling factors
per segment as well as the virtual marker positions. Additionally, a pelvis rotation
matrix was generated for each frame, since the pelvis functions as the relative
position of the model to the ground that is free to move in all directions.
2.2.2. Data Preparation and Hyperparameters. To generate the input for our network, each video frame was cropped and augmented. We use the pre-trained Faster
R-CNN [50] with ResNet-50 [22] backbone to extract a square bounding box of theperson in videos and resize it to 256 × 256 pixels as the input image size. During training, we apply data augmentation with scaling, rotation, translation andnoise to simulate occlusions similar to [51].
Our model was trained using the following hyperparameters and loss. For the
ResNeXt model, we use an Adam optimizer with weight decay [35] of 0.001 and abatch size of 64. The learning rate exponentially decays in two steps from 5 × 10−4
to 3.33 × 10−5 over 28 epochs and from 3.33 × 10−6 to 10−6 over 2 epochs. For bothsequential and convolutional networks, we set the hyperparameters with λ1 = 1.0,
λ2 = 2.0, λ3 = 0.1 and λ4 = 0.06 experimentally. Due to memory constraints, we
do not train convolutional and sequential models simultaneously, but in succession,
by first training the convolutional model and then refining predictions using thesequential model.
2.3. Software Tools. All training was conducted in python using the PyTorch library [47]. The pre-trained ResNext and FasterRCNN networks were obtained fromthe torchvision library [59]. All code for training and generation of ground truth
will be made available in a Github repository: https://github.com/bittnerma/
Direct3DKinematicEstimation. .
2.4. Data. We trained and tested D3KE on the BML-MoVi Database [19]. BMLMovi is an extensive motion capture and video dataset, it contains recordings of
90 actors that each perform 20 kinds of everyday movements as well as a random
one. Motions were captured using inertial measurement units as well as a Qualisys
optical motion capture system and videos were recorded using two computer-vision
cameras. For this study, we used recordings from the calibrated Point Gray cameras(PG1, PG2) during recording session F as the full set of optical markers was usedduring this session. In accordance with the anatomical plane that each camera isviewing during the initial T-Pose of the participants, we will refer to the camera
view captured by PG1 as the frontal- and PG2 as the sagittal camera view. For thegeneration of ground truth virtual markers, we use the 3D mesh representations of
the Qualisys data that is provided in the larger AMASS dataset [37]. For analysisof the data, we divided the BML-Movi database into 63 participants for training,
16 participants for the testing, and three participants for validation. This is common
practice in the supervised training of deep neural networks [64]. At training time,the validation set is used to evaluate the accuracy of kinematic estimation after
each training iteration on a portion of the data the network does not have access
to, to prevent overfitting on the training set.

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

11

3. Experiments
3.1. Experiment 1: Direct vs. Multi-Step Estimation. To evaluate the accuracy of our direct 3D kinematic estimation approach (D3KE) for joint angle
estimation, we compare its performance against multiple versions of the multistep approach.
3.1.1. Experiment 1-A: 3D Pose Based Kinematic Estimation. We first compare
our direct estimation of kinematics and a 3D pose estimation multi-step baseline.
To create a fair comparison between direct and multi-step estimations, we implement a custom multi-step approach (CMS) that is trained on the same data as our
direct approach. For the CMS, we combine a 3D human pose estimation methodwith subsequent musculoskeletal modeling in OpenSim. We modify the metric-scale
heatmaps [51] of the convolutional network to predict marker positions and SMPL
keypoint positions in the metric scale. As for D3KE, we exploit a sequence network
to re-fine marker positions at the target frame. More specifically, the convolutional network initially infers marker positions under a calibration pose (T-pose),and OpenSim utilizes the predicted marker data for body scaling, where the general musculoskeletal model is scaled to the participant’s body size. Re-fined marker
positions are then used to run inverse kinematics with the scaled musculoskeletal
model to obtain joint angles. The main difference between the CMS approach andD3KE is that CMS uses multiple steps to estimate the kinematics and is only supervised on the marker/pose estimation task, while D3KE is directly trained on the
kinematic estimation task; this way, we can compare direct vs. multi-step estimation of kinematics. We use multiple metrics for the comparison of D3KE and the
CMS. The mean per bony landmarks position error (MPBLPE) is used to evaluate
bony landmark positions. Bony landmarks are markers placed where bones areclose to the surface, such as the elbow. This metric is inspired by the mean per
joint position error (MPJPE) which is often used in 3D pose estimation. MPBLPE
first aligns estimations and ground truth at the root position and calculates theaverage Euclidean distance. We directly evaluate the body scale factors by the root
mean square error RMSEbody on the scalars predicted by the network. However,to present the results in a more intuitive format, we choose the axis along the longest
dimension in each body scale and convert the scale of the axis into millimeters andcalculate the mean absolute error (MAEbody ).
3.1.2. Experiment 1-B: 2D-Pose Based Kinematic Estimation. In the previous experiment, we evaluate the multi-step baseline with the 3D body pose estimation
method. However, the use of fully trained 2D pose estimation algorithms is common in kinematic estimation works [40, 45, 46]. Therefore, we conduct experiments
to compare our method and these 2D-based kinematic estimation methods. In contrast to our CMS method which estimates 3D pose from a single camera estimation,
2D pose estimation methods require at least 2 calibrated cameras for the estimation of 3D keypoints. The use of an additional camera to generate the pose could
be an advantage, which the CMS method does not have. We chose a naive implementation of the OpenPose algorithm [10, 25], which has extensively been used inrelated work [40, 45, 46]. Additionally, we test the MediaPipe implementation of
the blazepose algorithm [6], as a more modern 2D algorithm. MediaPipe is easy
to use since it is available as a python library, however, in contrast to OpenPose

12

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

it runs faster, allows for additional smoothing of its predictions, provides more key
points, and is labeled on different keypoint labels.
For the OpenPose and MediaPipe, we project the key points to 3D using the
BML-Movi camera parameters https://github.com/saeed1262/MoVi-Toolbox (accessed on 2 August 2022 ). For OpenPose, we connected missing points (due toself-occlusion) using linear interpolation. For MediaPipe, we chose the highest
model complexity (2) and set enabled smooth landmarks for continuous frames ofa video. For modeling and inverse kinematics, we follow the same steps as for the
CMS method only redefining the positions of markers on the OpenSim model to fit
the provided key points.
We compare against an average across both camera views for CMS and D3KE,
as MediaPipe and OpenPose need at least two cameras to work. We evaluate
performance based on mean absolute error M AEangle (◦ ), the standard deviationof errors SDangle (◦ ) and smoothness of the predictions as the mean velocity of theangle MVangle (◦ /s). The mean velocity error is calculated by the derivative of thelandmark position and joint angle data with respect to time.
(13)

M Vangle =n |st −st+1 |
X
∆t

t=0

n

with st an individual marker position at time t, ∆t is the amount of time between
time steps and n is the total number of timesteps.
3.2. Experiment 2: Sequential Network Variants. Since we have multiple
options for the sequential networks, we evaluate three to determine whether the
additional modeling of temporal dependencies in the data improves the accuracyof our method or not. For subsequent smoothing and reduction of self-occlusion
artifacts of the estimations, we test three different networks including LSTM [23],temporal convolutional networks (TCNs) [49], and a lifting Transformer [31] as
the sequential network. As smoothing is known to improve the accuracy of multistep approaches [40], we also evaluate combinations of our CMS model with thesesequential networks.
For the LSTM, we implement a bidirectional architecture with a hidden sizeof 128, three recurrent layers and a dropout probability of 0.1. For TCNs, wefollow [49] to exploit 243 frames as the receptive field and make the momentumof batch normalization decay from 0.1 to 0.001. For the lifting Transformer, we
use a hidden size of 256 and 8 parallel attention heads in the self-attention layer
and a channel size of 512 in the convolutional layer. Each sequential network is
trained with a sequence length of 243 frames and a batch size of 128 over 50 epochswith Adam optimizer [28]. The learning rate exponentially decays from 10−3 to
5 × 10−6 .
We use the same metrics for the comparison of individual network variants as
we used for the comparison of D3KE and CMS. In addition, we investigate thesmoothness of the predicted sequences, we estimate the mean velocity (MV) on
bony landmark positions and joint angles, denoted as MVBL and MVangle .
3.3. Experiment 3: Processing Speed. One important property of our proposed method for clinical applications is its processing speed. As applications forTOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

13

camera-based kinematic estimation should form an alternative to visual examinations in the future, it should ideally be able to run fast enough to estimate kinematics from video frames as fast as they are collected by a camera, mostly between
15 and 30 frames per second.
We compare the running time on Windows 10 with four core CPU, 52 GB RAM
and NVIDIA T4 GPU. We compare the CMS and D3KE method as they bothuse the same type of convolutional network. We choose the lifting Transformer
as the sequential architecture in both the CMS and D3KE. For the CMS method,
OpenSim is executed in parallel with four cores. Our report results in frames per
second.
3.4. Experiment 4: Generalization Performance. The goal of camera-based
kinematic estimation is ultimately to create tools for researchers and clinicians toanalyze and diagnose human movement, these tools should not discriminate between
different subjects and movements. We analyze whether our method generalizes todifferent subjects, movements, and joints.
To assess how well D3KE generalizes, we compare the estimates of the proposed
method to the ground truth on the time series of each of the 16 participants in thetest set with respect to the performed movement, the joint, the camera view andthe individual participant. For this test, we use the best performing model fromexperiment 1 with the lifting transformer.
For all time series of joint angles, mean absolute error (MAE) and Pearson’s
correlation coefficient (ρ) were calculated between the estimation from D3KE andthe ground truth.
Central tendencies in the data are reported as a median and interquartile rangeof MAE, RMSE and ρ, as the data are not normally distributed, as assessed throughvisual inspection and confirmed by the Shapiro-Wilk test. For completion, meanand standard deviation are also reported. The absolute values of ρ were categorized
as weak, moderate, strong and excellent for ρ ≤ 0.35, 0.35 < ρ ≤ 0.67, 0.67 < ρ ≤0.90 and 0.90 < ρ, respectively [56].
3.5. Software and Tools. All data analysis was conducted in python 3 [58] using
the pandas library to generate descriptive statistics, SciPy library for the calculationof MAE, RMSE and Pingouin library for the calculation of ρ.
4. Results
4.1. Direct vs. Multi-Step Estimation.
4.1.1. A: 3D Pose Based Kinematic Estimation. As shown in Table 1, D3KE has
better performance than CMS methods in terms of joint angles and body scales,
and these two factors are the key to kinematic estimation. Significantly, D3KE
reduces 37.4% of errors on MAEangle when comparing the CMS method with theTransformer architecture. Although the proposed method has a slightly larger MPBLPE than the CMS, this metric is not related to the kinematic estimation and is
only used as one of the losses during training in the proposed method (e.g., MPBLBE is not minimized). This indicates a gain in accuracy when directly estimating
kinematics from the video instead of the multi-step approach of first estimating poseand then estimating kinematics.

14

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

Table 1. Comparison of bony landmarks position (MPBLPE),
body scales (MAEbody) and joint angles (MAEangle) between theestimation of and the ground truth across all participants, movements, joints and camera views. For the custom multi-step approach (CMS) as well as our proposed method, we compare convolutional networks with different temporal networks. All versions of
the proposed method show superior performance for the predictionof body scales and joint angle estimation. All CMSs show superior performance in estimating marker positions. Each method
group shows better performance for the task it was optimized for,highlighting the importance of direct optimization. Bold numbers
indicate the best performance.

D3KE

CMS

MPBLPE(mm)

MAEbody (mm)MAEangle (◦ )Convolutional
Conv.+ LSTM
Conv.+ TCNs
Conv.+
Transformer

37.78
37.61
38.06
36.98

6.07
5.97
5.93
5.90

3.58
3.57
3.54
3.54

Convolutional
Conv.+ LSTM
Conv.+ TCNs
Conv.+
Transformer

35.04
33.74
34.52
34.00

6.25
-

5.89
5.79
5.82
5.66

In Table 2, we list RMSE and MAE for body scales of selected segments. The results show that the CMS performs better than D3KE on lower limbs, and D3KE
performs better than the CMS on upper limbs. The CMS and the proposed method
have comparable performance in scale estimation of the pelvis and lower limbs.
Table 2. Errors in scaling factors of the proposed method andthe baseline compared against the ground truth. D3KE shows
better performance for the upper extremities and slightly worse
performance for the lower extremities.
RMSEbody (MAEbody (mm))
pelvisfemur
tibia
humerusulna
radius

CMS

D3KE

0.090 (9.58)
0.073 (10.55)
0.060 (9.82)
0.102 (14.55)
0.395 (24.91)
0.395 (23.60)

0.091 (9.82)
0.091 (22.21)
0.102 (35.00)
0.068 (9.41)
0.075 (11.59)
0.075 (10.98)

4.1.2. B: 2D Pose Based Kinematic Estimation. The results of the comparison of
our proposed method, CMS, OpenPose, and MediaPipe are shown in Table 3. We

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

15

find that algorithms trained on noisy labels, that use fewer key points perform worse
than ours. We see a clear difference between the unsmoothed OpenPose estimations
and the smoothed MediaPipe estimations in the mean velocity of the estimations.
Our proposed method D3KE still performs better showing that even in an ideal scenario (CMS, i.e., no noise in the labels, enough markers, same distribution trainingdata) direct estimation is preferable.
Table 3. Comparison of popular pose estimation algorithms to
D3KE. As OpenPose and MediaPipe require multiple cameras tocreate 3D keypoints, we compare against the average of both camera views for CMS and D3KE. CMS shows better performance than
OpenPose and MediaPipe and D3KE shows the overall best performance. Indicating that direct estimation is preferable to (naive)
implementations of multi-step methods.
M AEangle (◦ )SDangle (◦ )M Vangle (◦ /s)

16.98
10.60
5.11
3.41

25.91
18.80
10.27
6.05

75.15
37.15
15.74
13.57

OpenPoseMediaPipe
CMS
D3KE

4.2. Sequential Network Variants. Table 1 also shows the results of different
sequential networks for smoothing of the predictions. Although the convolutionalmodel by itself already has good performance in joint angle and scale factor estimation, using temporal smoothing can additionally reduce the estimation error.
The results of our investigation to reduce the noise in the estimations using
temporal smoothing are shown in Table 4. The result shows that all temporalmodels can improve the smoothness of the sequence. The LSTM achieves the best
performance on MVBL and MVangle among all temporal models, this is contrary to
the results in Table 1, in which using a Transformer as the sequential model yielded
the best results.
Table 4. The mean velocity errors for bony landmarks MVBL andjoint angles MVangle , lower values indicate smoother estimations.
Adding a sequential model most probably improves the continuityof estimations.

Convolutional modelConv. + LSTM
Conv. + TCNs
Conv. + Transformer

MVBL (mm/s)

MVangle (◦ /s)

378.01243.23245.35262.82

21.7
12.19
12.29
13.57

4.3. Processing Speed. Our proposed method achieves 31.96 fps with a batch
size of 256, as shown in Table 5. Since the skeletal-model layer must traverse bodysegments in the level order, our proposed method is slower than the CMS for a

16

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

batch size of 1. However, the support of mini-batch computation in the skeletalmodel layer allows D3KE to run faster than the CMS. Showing that our method
can reach video framerate speeds on a competent GPU.

Table 5. Comparison of processing speed in FPS of D3KE andthe baseline for multiple images or ’batches’ in parallel. OpenSim
does show little change in processing speed for increasing batch
sizes. The proposed method achieves framerate speeds for batchesof 256 images, allowing it to analyze images as fast as a common
webcam or mobile phone camera collects them. Bold numbers
indicate best performance.
Batch
Size

1

16

64

128

256

D3KE
CMS

0.92
7.51

8.78
8.36

20.94
8.43

28.25
8.44

31.96
8.35

4.4. Generalization Performance. Figure 4 shows that both CMS and D3KE
have relatively little variation across different movements and different participants,
yet larger variations across individual joints. This is also reflected in median MAEs
and ρ per joint (Table 6), with median MAEs for joints varying within a range 3.8◦
for joints, while movements and participants vary under 1◦ . It shows that D3KE
generalizes well to different participants and movements.

Figure 4. Mean absolute error for predicted joint angles per joint,
movement and subset. Across these groups, D3KE shows less variation compared to the CMS. The low variations indicate that D3KE
is suitable for use on different participants and movements.

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

17

Table 6. Median and inter-quartile ranges (IQR) of joint angles
per joint, movement and participant. MAE, RMSE and correlation are calculated over individual frames, Medians and IQR arereported due to the skewed distribution of results. Within eachgroup, both camera views show similar errors. Joint angles show
the highest error and highest spread of values of all groupings.
D3KE generalizes well to different movements, participants andcamera views.
Group

Camera View

Joint

FrontalSagittal
FrontalSagittal
FrontalSagittal

Movement
Participant

MAE (◦ )Median IQR
2.13
2.14
1.85
1.91
1.76
1.84

3.80
3.03
0.63
0.46
0.53
0.28

RMSE (◦ )Median IQR
2.54
2.55
2.19
2.30
2.03
2.14

3.96
3.49
0.84
0.65
0.66
0.35

ρ
Median IQR
0.77
0.73
0.76
0.74
0.77
0.74

0.16
0.21
0.11
0.11
0.04
0.04

4.5. Qualitative Results. We visualize the estimation of musculoskeletal models
from our proposed method with the Transformer architecture in Figure 5. We also
show the comparison between estimation and ground truth of the left knee angle as
an example of joint angle estimation quality. We took the average body scales of thepredicted sequence of scaling factors to scale the model and visualize it in OpenSim
using the predicted joint angles as inputs. From the figure, we can see that the
proposed method can achieve results that are in agreement with the single-view
input video.

18

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

Figure 5. Qualitative results of D3KE from a ’sitting-down’movement in the BML-Movi dataset. The top row shows selected
frames throughout the movement. The middle row shows different
poses of the ground truth skeletal model throughout the movement(cyan) and the skeletal model (white) based on D3KE’s estimation.
The bottom row shows the changes in flexion/extension of the left
knee throughout the movement with blue being the predicted andorange being the ground truth angle.

5. Discussion
In summary, we compared a direct approach of estimating joint angles fromvideo images to the more traditional multi-step approach found in most recent
works. The traditional method first estimates key points from a video of a subject,then calculates joint angles using a (musculo)skeletal model through an inversekinematics process. We developed a method consisting of a convolutional neuralnetwork and a sequential network both including a specialized layer that performs
kinematic transforms of a (musculo)skeletal model and allows for direct optimization of the predicted joint angles (D3KE) and treats the prediction of key points
only as an auxiliary task. We compared our direct estimation approach against
naive implementations of often used algorithms in the related literature, as well asa self-implemented custom multi-step approach (CMS) that is trained on the samedata as our direct approach. We show that direct estimation of kinematics yields
higher accuracy in predicted joint angles compared to the traditional multi-step
approach. Our results indicate that direct estimation can help the future development of algorithms for fast and accessible kinematic analysis for researchers andclinicians.
5.1. Direct vs. Multi-Step Estimation.
5.1.1. 3D-Pose Kinematic Estimation. To compare direct estimation vs. multi-stepestimation, we compared our D3KE method against a 3D-pose based multi-step
approach (CMS) with comparable network architecture and trained on the same

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

19

training data. Compared to the CMS, our proposed method improves the accuracyof joint angle estimation. For all model combinations, we can see an improvementof about 35% in accuracy for the estimation of joint angles. Our results support
the feasibility of our proposed method. It delivers improvements due to directly
optimizing the predicted joint angles and scaling factors while using the pose estimates only as an auxiliary task. The auxiliary task effectively imposes a constraint
on the network estimation. We show that direct optimization is preferable to themulti-step approach when using videos from a single-camera view. We expect thatusing additional specialized layers, a network might be able to directly optimize forindividual muscle forces with comparable accuracy from a monocular video.
5.1.2. 2D-Pose Kinematic Estimation. We compared our direct approach (D3KE)
and the self-trained multi-step approach (CMS) against two multi-step approaches
commonly used in the literature. Compared to the more traditional implementations of the OpenPose and MediaPipe algorithms, both our proposed and our CMS
method show superior performance. From Tables 4 and 6, we see that estimations
from D3KE are far smoother compared to the traditional methods. This is likely
due to multiple reasons. The predicted key points of OpenPose suffer from systematic errors due to inaccuracies in their training data [13], which can explain the dropin performance, for MediaPipe the accuracy of labels in their training data is notknown as their paper only states that their annotators were human [6], not whether
they had expertise in labeling anatomical key points. The lack of smoothing for thepredicted OpenPose key points can also contribute to its overall worse performance.
MediaPipe, which uses internal smoothing, shows better results in comparison. In
general, we expected worse performance from OpenPose and MediaPipe as they
only predict 18 or 33 key points respectively, while we supervise a total of 77. Bothtraditional methods predict keypoints representing joint centers and not markers
on body segments, this makes it hard to distinguish rotations between different
body segments during the musculoskeletal modeling step.

Figure 6. Qualitative comparison of predicted angles of the leftknee, from a ’sitting-down’ movement in the BML-Movi dataset.
While OpenPose shows by far the noisiest estimation, the smoothing of the MediaPipe estimation is clear, our proposed method
and implemented CMS work best, probably due to the restrictionof additional markers.

20

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

5.2. Network Variants. We compared three commonly used sequential networks
to improve our estimation. We show that all sequential networks improve ourestimation accuracy. One possible explanation for this is that the network is better
able to handle ‘self-occlusion’ artifacts. The estimation of a 3D-pose from a single
camera is an ill-posed problem as a 3D point projected to a 2D image can originate
from any position along the ray(s) that fall on the image sensor and form thecorresponding pixel. This can lead to self-occlusion, such as the torso and left arm
occluding the right arm during a right arm swing in frames recorded from the left
sagittal view. During self-occlusion, it is difficult for frame-based networks to makea good estimate as they lack temporal information of previous angles of the arm to
extrapolate from. Sequential networks on the other hand have access to temporal
information, which can allow for more accurate estimations. Although we only see
a slight increase 0.001◦ in MAE of the joint angle estimation, we can see a clear
improvement of the sequential models in the smoothness of the predicted angles
Table 4. This might be due to the network learning to interpolate motions during
occurrences of self-occlusion.
5.3. Processing Speed. Compared to the multi-step baseline, CMS, our D3KE
approach shows increased calculation speeds for larger batch sizes. Both CMS
and D3KE make use of the same ResNeXt50 architecture, which should show approximately the same performance increase with increasing batch sizes for bothmethods. D3KE could be expected to be slower, as it also has the additional time
cost of calculating the pose from the estimated kinematics in the skeletal model
layer. However, due to its multi-step nature, CMS has to perform an additional
inverse kinematics calculation. This calculation seems to form a bottleneck in theprocessing speed of the CMS approach restricting it to a framerate of 8 fps. Other
multi-step algorithms will most likely encounter the same problem. In the case of
OpenPose, which runs at about 4 fps [6], even lower frame rates can be expected
for a complete pipeline. This shows the advantage in the processing time of ourdirect approach.
For a method to be usable in everyday life, it should be reasonably fast in
running. Processing speeds allowing a method to run between 15 and 30 frames
per second are favorable, as they show that a method can process a video as fast as
its frames are collected. However, our results might not directly translate to every
real-world scenario. To process multiple images simultaneously as batches, D3KE
currently requires GPUs that are not available in mobile devices, which prevents
it from beingportable. In addition, we use the Faster R-CNN object detection
network to crop our images. This step was not included in the processing speed
evaluation, as it is highly dependent on the chosen object-detection algorithm.
However, with inference speeds of 12fps, the Faster R-CNN object detection would
form a bottleneck in applying our method in real-time applications. Given the
speed of development in the field of object detection, Faster R-CNN can by now
be regarded as an old algorithm, and newer and faster object detectors should be
used instead. The YoloV7 algorithm [61], which performs object detection at up to
286 fps could be considered. In general, the current architecture is not optimized
for speed or a specific technology and we are using off-the-shelf, fairly standard
convolutional and sequential architectures. For these architectures, smaller andfaster alternatives might be found in the future. When optimizing for all thesepoints, we predict the proposed method could run on mobile devices within a few

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

21

years, effectively enabling a 3D kinematic analysis instrument to become available
for everyone with a mobile phone or tablet.
5.4. Generalization Performance. Our method generalizes well on the tested
data. As shown in Figure 4 and Table 6, the estimation variations across participants and movements are small. We can conclude that our method shows theability to generalize to different camera views, participants, and performed movements, within the tested dataset. Our results indicate that D3KE could be generally
applied to a variety of people and movements, including clinical and sports applications, e.g., physiotherapists and athletes, when trained on sufficient additional
data.
Although we show good generalization performance on the BMLMovi database,
it is difficult to estimate how well our method will generalize in a real-world scenario.
In machine learning settings, training data is often not representative of the taskof the network in the real world [5] and can introduce biases if applied to scenariosthat are very different from the one represented in the training data. Unfortunately,
there is currently a lack of deep learning datasets for kinematic analysis [52, 40,
13, 60]. In addition, while the BML-Movi database is excellent for training neuralnetworks due to the large number of participants performing movements and thediversity of execution styles, it might be not extensive enough to train a network forbiomedical applications in the real world. However, to evaluate the current method
fully, such an extensive dataset would be necessary. In general, we expect a dropin accuracy when our method is applied to a scenario different from the BMLMovi
database. As we train on just two calibrated cameras, we expect our method to be
most vulnerable to alternative camera positions, that do not show people in either
frontal or sagittal view. Future research should investigate the stability of direct
estimation methods when applied to data that differs significantly from the training
data.
5.5. Future Work. To improve the accuracy of the algorithm and provide further
insight into the strengths and weaknesses of monocular joint angle estimation, a
new dataset with dedicated annotation is needed. A dataset specifically designed
for the estimation of joint angles and/or kinetics could improve the accuracy of thealgorithm. This dataset could be established with a large number of camera views,
and top-down views for better estimation of movements in the transverse plane,
where participants perform movements that exercise the full ROMs of individual
joints including upper extremities, as well as movements that are relevant for healthcare professionals such as physiotherapy exercises and other clinical tests. In addition, the inclusion of abnormal movement patterns could give better insights into
the clinical relevance of newly developed methods.
Transfer learning could be explored to apply 3DKE in settings where little training data are available. Vdeos that are very different from the BMLMovi training
data, such as people wearing more clothes, are in different surroundings, or are
filmed from a different camera view, will, most probably, yield worse accuracy thanshown in this paper. Transfer learning of a pre-trained D3KE on a minimal portionof a dataset could be investigated as an alternative to the time-consuming collectionof a novel dataset.
The capabilities of D3KE as an adapter for kinetic analysis of a movement in
OpenSim could be explored. Given data similar to BMLMovi or successful transfer

22

TOWARDS SINGLE CAMERA HUMAN 3D-KINEMATICS

learning on relevant data beforehand, our method provides an easy way to skip thetedious steps of scaling and running inverse kinematics on an MSM. This enables thequick generation of MSMs for kinetic analysis from just a single video. Even if thiskinematic estimation comes at the cost of reduced accuracy, it could provide coarse
insights into collected data, which can later be confirmed through finer analysis
with the manually scaled MSMs.
D3KE could be made more generally applicable if the underlying model of the
Skeletal-model layer would not be fixed. Currently, the underlying model is fixedin the Skeletal-model layer. Future iterations could explore combinations of thePytorch and OpenSim python libraries to allow training a network on a self-definedmodel or allowing a pre-trained model to be refined through transfer learning for,e.g., only estimation of joint angles around the shoulder.
Existing Explainable AI tools should be applied to better understand the innerworkings of D3KE. Deep neural networks are capable of high accuracy estimation,
because of their ability to break down highly complex tasks into simpler tasks [2],but understanding what these simpler tasks are is non-trivial. Research in Explainable AI has generated tools and frameworks that allow one to better understandthe basis of the final predictions of a network [53]. Applying these tools could help
users and researchers alike to better understand the biases and limitations of our
method. D3KE can still predict the joint angles even if these joints are occluded;
this means it must make assumptions. What these assumptions are and how theycame to be are important to estimate the trustworthiness of this algorithm in areal-world scenario.
6. Conclusions
In this paper, we present a novel end-to-end neural network for the estimationof segment joint angles of the human body. Compared to the previous method,
we directly regress to the joint angle and scale for individual segments from the
input video. We trained our method from scratch on the BML-Movie database andcompared it against a 3D pose estimation method on which we used the inverse
kinematics tool of OpenSim to obtain the kinematics.
We conclude that using direct estimation of joint angles is preferable in a single
camera setting, as it is more accurate compared to the common approach of fitting
an estimated pose to a musculoskeletal model and performing inverse kinematics.
By allowing the network to directly optimize for the joint angles and scaling factors,
our method is less prone to errors in the key point labels used to predict key point
location for pose estimation. In addition, the use of a sequential model is importantwhen designing a neural network architecture for kinematic estimation, as it allows
to smooth predictions over time to create better estimates of limb position and joint
angles during self-occlusion.
While using deep learning for biomedical solutions is still in its infancy, the presented method shows that training networks from scratch for specialized tasks isa viable way to estimate joint angles from a single camera video. With further
advancements in the underlying algorithms as well computational performance, we
predict that the methodology we have presented will assist biomedical and clinic
practitioners to measure and monitor human movement in the near future.
Funding: This work was supported by the Dutch Research Council (NWO) under
the Citius Altius Sanius Perspective Program P16-28 Project 4.

REFERENCES

23

Aknowledgements: The authors would like to thank Lisa Noteboom for her
feedback as well as Marco Hoozemans and Dirkjan Veeger for guidance and insightduring our bi-weekly meetings.
Conflict of Interest: The authors declare no conflict of interest.
Abbreviations: The following abbreviations are used in this manuscript:
CMS . . . . . . . . . . . . Custom multi-stage approach
D3KE . . . . . . . . . . . Direct 3D kinematic estimation
IQR . . . . . . . . . . . . . Interquartile Range
MAE . . . . . . . . . . . . Mean absolute error
MMC . . . . . . . . . . . Markerless motion capture
MPBLPE . . . . . . . Mean per bony landmark error
MSM . . . . . . . . . . . . Musculoskeletal model
MVE . . . . . . . . . . . . Mean velocity error
OMC . . . . . . . . . . . . Optical Motion Capture
PCC . . . . . . . . . . . . . Pearson correlation coefficient
RMS . . . . . . . . . . . . Root mean square error
ROM . . . . . . . . . . . . Range of motion
SD . . . . . . . . . . . . . . . Standard Deviation
TCN . . . . . . . . . . . . Temporal Convolutional Network
References
[1]

[2]

[3]

[4]

[5]

Mazen Al Borno, Johanna O’Day, Vanessa Ibarra, James Dunne, Ajay Seth,
Ayman Habib, Carmichael Ong, Jennifer Hicks, Scott Uhlrich, and Scott
Delp. “OpenSense: An open-source toolbox for inertial-measurement-unitbased measurement of lower extremity kinematics over long durations”. In:
Journal of NeuroEngineering and Rehabilitation 19.1 (Feb. 2022), p. 22. issn:1743-0003. doi: 10.1186/s12984- 022- 01001- x. url: https://doi.org/
10.1186/s12984-022-01001-x (visited on 10/27/2022).
Zeyuan Allen-Zhu and Yuanzhi Li. Backward Feature Correction: How Deep
Learning Performs Deep Learning. arXiv:2001.04413 [cs, math, stat]. Mar.
2021. doi: 10.48550/arXiv.2001.04413. url: http://arxiv.org/abs/
2001.04413 (visited on 12/07/2022).
FRANK C. ANDERSON and MARCUS G. PANDY. “A Dynamic Optimization Solution for Vertical Jumping in Three Dimensions”. In: Computer
Methods in Biomechanics and Biomedical Engineering 2.3 (1999). PMID:
11264828, pp. 201–231. doi: 10.1080/10255849908907988. eprint: https:/ / doi . org / 10 . 1080 / 10255849908907988. url: https : / / doi . org / 10 .
1080/10255849908907988.
Frank C. Anderson and Marcus G. Pandy. “Dynamic Optimization of Human Walking ”. In: Journal of Biomechanical Engineering 123.5 (May 2001),
pp. 381–390. issn: 0148-0731. doi: 10 . 1115 / 1 . 1392310. eprint: https ://asmedigitalcollection.asme.org/biomechanical/article-pdf/123/
5/381/5590682/381\_1.pdf. url: https://doi.org/10.1115/1.1392310.
Joshua Attenberg, Panos Ipeirotis, and Foster Provost. “Beat the Machine:
Challenging Humans to Find a Predictive Model’s “Unknown Unknowns””.
In: Journal of Data and Information Quality 6.1 (Mar. 2015), 1:1–1:17. issn:
1936-1955. doi: 10 . 1145 / 2700832. url: https : / / doi . org / 10 . 1145 /
2700832 (visited on 12/11/2022).

24

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

REFERENCES

Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan
Zhang, and Matthias Grundmann. BlazePose: On-device Real-time Body Pose
tracking. arXiv:2006.10204 [cs]. June 2020. doi: 10 . 48550 / arXiv . 2006 .
10204. url: http://arxiv.org/abs/2006.10204 (visited on 10/21/2022).
Alexander G Bruno, Katelyn Burkhart, Brett Allaire, Dennis E Anderson,and Mary L Bouxsein. “Spinal Loading Patterns From Biomechanical Modeling Explain the High Incidence of Vertebral Fractures in the Thoracolumbar
Region”. In: Journal of Bone and Mineral Research 32.6 (2017), pp. 1282–
1290. doi: https : / / doi . org / 10 . 1002 / jbmr . 3113. eprint: https : / /asbmr . onlinelibrary . wiley . com / doi / pdf / 10 . 1002 / jbmr . 3113. url:
https://asbmr.onlinelibrary.wiley.com/doi/abs/10.1002/jbmr.3113.
Alexander G. Bruno, Mary L. Bouxsein, and Dennis E. Anderson. “Development and Validation of a Musculoskeletal Model of the Fully Articulated Thoracolumbar Spine and Rib Cage”. In: Journal of Biomechanical Engineering
137.8 (June 2015). 081003. issn: 0148-0731. doi: 10.1115/1.4030408. eprint:
https : / / asmedigitalcollection . asme . org / biomechanical / article pdf/137/8/081003/6092282/bio\_137\_08\_081003.pdf. url: https://doi.org/10.1115/1.4030408.
Katelyn Burkhart, Daniel Grindle, Mary L. Bouxsein, and Dennis E. Anderson. “Between-session reliability of subject-specific musculoskeletal modelsof the spine derived from optoelectronic motion capture data”. In: Journal of Biomechanics 112 (2020), p. 110044. issn: 0021-9290. doi: https :/ / doi . org / 10 . 1016 / j . jbiomech . 2020 . 110044. url: https : / / www .
sciencedirect.com/science/article/pii/S0021929020304681.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.
2019. arXiv: 1812.08008.
Yu Cheng, Bo Yang, Bo Wang, and Robby T. Tan. 3D Human Pose Estimation using Spatio-Temporal Networks with Explicit Occlusion Training. 2020.
arXiv: 2004.11822 [cs.CV].
Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T. Tan. “OcclusionAware Networks for 3D Human Pose Estimation in Video”. In: Proceedings ofthe IEEE/CVF International Conference on Computer Vision (ICCV). Oct.
2019.
Neil J. Cronin. “Using deep neural networks for kinematic analysis: Challenges and opportunities”. en. In: Journal of Biomechanics 123 (June 2021),
p. 110460. issn: 0021-9290. doi: 10.1016/j.jbiomech.2021.110460. url:
https://www.sciencedirect.com/science/article/pii/S0021929021002402(visited on 03/15/2022).
U. Della Croce, A. Cappozzo, and D. C. Kerrigan. “Pelvis and lower limb
anatomical landmark calibration precision and its propagation to bone geometry and joint angles”. en. In: Medical & Biological Engineering & Computing
37.2 (Mar. 1999), pp. 155–161. issn: 1741-0444. doi: 10.1007/BF02513282.
url: https://doi.org/10.1007/BF02513282 (visited on 06/03/2022).
S.L. Delp, J.P. Loan, M.G. Hoy, F.E. Zajac, E.L. Topp, and J.M. Rosen. “Aninteractive graphics-based model of the lower extremity to study orthopaedic
surgical procedures”. In: IEEE Transactions on Biomedical Engineering 37.8
(1990), pp. 757–767. doi: 10.1109/10.102791.

REFERENCES

[16]

[17]

[18]

[19]

[20]

[21]

[22]
[23]

[24]

[25]
[26]

[27]

25

Scott L. Delp, Frank C. Anderson, Allison S. Arnold, Peter Loan, Ayman
Habib, Chand T. John, Eran Guendelman, and Darryl G. Thelen. “OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations ofMovement”. In: IEEE Transactions on Biomedical Engineering 54.11 (2007),
pp. 1940–1950. doi: 10.1109/TBME.2007.901024.
Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. “RMPE: Regional
Multi-Person Pose Estimation”. In: 2017, pp. 2334–2343. url: https : / /openaccess.thecvf.com/content_iccv_2017/html/Fang_RMPE_Regional_
Multi-Person_ICCV_2017_paper.html (visited on 07/18/2022).
Mickael Fonseca, Stéphane Armand, Raphaël Dumas, Fabien Leboeuf, Mariette Bergere, and João Cândido. “The Conventional Gait Model’s Sensitivity
to Lower-limb Marker Placement”. In: (2022).
Saeed Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje. “MoVi: A large
multi-purpose human motion and video dataset”. en. In: PLOS ONE 16.6
(June 2021). Publisher: Public Library of Science, e0253157. issn: 1932-6203.
doi: 10 . 1371 / journal . pone . 0253157. url: https : / / journals . plos .
org/plosone/article?id=10.1371/journal.pone.0253157 (visited on
12/01/2021).
R. Gonzalez, T. Buchanan, and S. Delp. “How muscle architecture and moment arms affect wrist flexion-extension moments.” In: Journal of biomechanics 30 7 (1997), pp. 705–12.
X. Gu, F. Deligianni, B. Lo, W. Chen, and G.Z. Yang. “Markerless gait analysis based on a single RGB camera”. In: 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN). ISSN:
2376-8894. Mar. 2018, pp. 42–45. doi: 10.1109/BSN.2018.8329654.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual
Learning for Image Recognition. 2015. arXiv: 1512.03385 [cs.CV].
Sepp Hochreiter and Jürgen Schmidhuber. “Long Short-Term Memory”. In:
Neural Comput. 9.8 (Nov. 1997), pp. 1735–1780. issn: 0899-7667. doi: 10.
1162/neco.1997.9.8.1735. url: https://doi.org/10.1162/neco.1997.
9.8.1735.
K. R. S. Holzbaur, W. M. Murray, and S. L. Delp. “A model of the upper
extremity for simulating musculoskeletal surgery and analyzing neuromuscular control”. In: Annals of biomedical engineering 33.6 (June 2005), pp. 829–
840. issn: 0090-6964. doi: 10 . 1007 / s10439 - 005 - 3320 - 7. url: https ://doi.org/10.1007/s10439-005-3320-7.
Hzzone. pytorch-openpose. https://github.com/Hzzone/pytorch-openpose.
3021.
Robert M. Kanko, Elise K. Laende, Gerda Strutzenberger, Marcus Brown,
W. Scott Selbie, Vincent DePaul, Stephen H. Scott, and Kevin J. Deluzio.
“Assessment of spatiotemporal gait parameters using a deep learning algorithmbased markerless motion capture system”. en. In: Journal of Biomechanics
122 (June 2021), p. 110414. issn: 0021-9290. doi: 10 . 1016 / j . jbiomech .
2021.110414. url: https://www.sciencedirect.com/science/article/
pii/S0021929021001949 (visited on 10/27/2022).
Lukasz Kidziński, Bryan Yang, Jennifer L. Hicks, Apoorva Rajagopal, Scott
L. Delp, and Michael H. Schwartz. “Deep neural networks enable quantitative

26

[28]
[29]

[30]

[31]

[32]

[33]

[34]

[35]
[36]

[37]

[38]

REFERENCES

movement analysis using single-camera videos”. en. In: Nature Communications 11.1 (Aug. 2020). Bandiera abtest: a Cc license type: cc by Cg type:
Nature Research Journals Number: 1 Primary atype: Research Publisher:
Nature Publishing Group Subject term: Data processing;Diagnostic markers;Machine learning;Movement disorders Subject term id: data-processing;diagnosticmarkers;machine-learning;movement-disorders, p. 4054. issn: 2041-1723. doi:
10.1038/s41467-020-17807-z. url: https://www.nature.com/articles/
s41467-020-17807-z (visited on 11/30/2021).
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 2017. arXiv: 1412.6980 [cs.LG].
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks”. In: Communications of theACM 60.6 (2017). Publisher: AcM New York, NY, USA, pp. 84–90.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. “Deep learning”. en. In:
Nature 521.7553 (May 2015). Number: 7553 Publisher: Nature PublishingGroup, pp. 436–444. issn: 1476-4687. doi: 10 . 1038 / nature14539. url:
https://www.nature.com/articles/nature14539 (visited on 12/07/2022).
Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, and Pichao Wang. Lifting
Transformer for 3D Human Pose Estimation in Video. 2021. arXiv: 2103.
14304 [cs.CV].
Rijun Liao, Shiqi Yu, Weizhi An, and Yongzhen Huang. “A model-based gait
recognition method with body pose and human prior knowledge”. en. In:
Pattern Recognition 98 (Feb. 2020), p. 107069. issn: 0031-3203. doi: 10 .
1016/j.patcog.2019.107069. url: https://www.sciencedirect.com/
science/article/pii/S003132031930370X (visited on 06/10/2022).
Ruixu Liu, Ju Shen, He Wang, Chen Chen, Sen-ching Cheung, and Vijayan K.
Asari. Enhanced 3D Human Pose Estimation from Videos by using AttentionBased Neural Network with Dilated Convolutions. 2021. arXiv: 2103.03170[cs.CV].
Matthew Loper, Naureen Mahmood, and Michael J. Black. “MoSh: Motion
and Shape Capture from Sparse Markers”. In: ACM Trans. Graph. 33.6 (Nov.
2014). issn: 0730-0301. doi: 10.1145/2661229.2661273. url: https://doi.
org/10.1145/2661229.2661273.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization.
2019. arXiv: 1711.05101 [cs.LG].
T. -W. Lu and J. J. O’Connor. “Bone position estimation from skin marker
co-ordinates using global optimisation with joint constraints”. en. In: Journalof Biomechanics 32.2 (Feb. 1999), pp. 129–134. issn: 0021-9290. doi: 10 .
1016/S0021- 9290(98)00158- 4. url: https://www.sciencedirect.com/
science/article/pii/S0021929098001584 (visited on 10/27/2022).
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll,
and Michael J. Black. “AMASS: Archive of Motion Capture as Surface Shapes”.
In: International Conference on Computer Vision. Oct. 2019, pp. 5442–5451.
Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh
N. Murthy, Mackenzie Weygandt Mathis, and Matthias Bethge. “DeepLabCut: markerless pose estimation of user-defined body parts with deep learning”. en. In: Nature Neuroscience 21.9 (Sept. 2018), pp. 1281–1289. issn:

REFERENCES

[39]

[40]

[41]

[42]
[43]

[44]

[45]

[46]

[47]

27

1097-6256, 1546-1726. doi: 10 . 1038 / s41593 - 018 - 0209 - y. url: http ://www.nature.com/articles/s41593-018-0209-y (visited on 05/04/2022).
Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin,
Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. “VNect”. In: ACM Transactions on Graphics 36.4 (July 2017),
pp. 1–14. issn: 1557-7368. doi: 10 . 1145 / 3072959 . 3073596. url: http ://dx.doi.org/10.1145/3072959.3073596.
Laurie Needham, Murray Evans, Darren P. Cosker, and Steffi L. Colyer. “Can
Markerless Pose Estimation Algorithms Estimate 3D Mass Centre Positions
and Velocities during Linear Sprinting Activities?” en. In: Sensors 21.8 (Jan.
2021). Number: 8 Publisher: Multidisciplinary Digital Publishing Institute,p. 2889. doi: 10.3390/s21082889. url: https://www.mdpi.com/14248220/21/8/2889 (visited on 12/01/2021).
Laurie Needham, Murray Evans, Darren P. Cosker, Logan Wade, Polly M.
McGuigan, James L. Bilzon, and Steffi L. Colyer. “The accuracy of several
pose estimation methods for 3D joint centre localisation”. en. In: Scientific
Reports 11.1 (Oct. 2021). Number: 1 Publisher: Nature Publishing Group,
p. 20673. issn: 2045-2322. doi: 10 . 1038 / s41598 - 021 - 00212 - x. url:
https : / / www . nature . com / articles / s41598 - 021 - 00212 - x (visited on
03/16/2022).
Aiden Nibali, Zhen He, Stuart Morgan, and Luke Prendergast. 3D Human
Pose Estimation with 2D Marginal Heatmaps. 2018. arXiv: 1806.01484 [cs.CV].
Lisa Noteboom, Marco J M Hoozemans, H E J Veeger, and Frans C T Van Der
Helm. “Feasibility and validity of a single camera CNN driven musculoskeletal
model for muscle force estimation during upper extremity strength exercises:
Proof-of-concept”. eng. In: Frontiers in sports and active living 4 (Jan. 2022),
p. 994221. issn: 2624-9367. doi: 10.3389/fspor.2022.994221. url: https://europepmc.org/articles/PMC9541110 (visited on 10/21/2022).
OpenSim. Modification of Wrist Model to include all the movements of thefingers. [Online] SimTK. Feb. 2018. url: https://simtk.org/projects/
moving-fingers.
David Pagnon, Mathieu Domalain, and Lionel Reveret. “Pose2Sim: An Endto-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness”. en. In: Sensors 21.19 (Jan. 2021). Number: 19 Publisher: Multidisciplinary Digital Publishing Institute, p. 6530. issn: 1424-8220. doi: 10.3390/
s21196530. url: https://www.mdpi.com/1424-8220/21/19/6530 (visitedon 03/15/2022).
David Pagnon, Mathieu Domalain, and Lionel Reveret. “Pose2Sim: An Endto-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy”.
en. In: Sensors 22.7 (Jan. 2022). Number: 7 Publisher: Multidisciplinary Digital Publishing Institute, p. 2712. issn: 1424-8220. doi: 10.3390/s22072712.
url: https://www.mdpi.com/1424-8220/22/7/2712 (visited on 05/13/2022).
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury,
Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, and Luca
Antiga. “Pytorch: An imperative style, high-performance deep learning library”. In: Advances in neural information processing systems 32 (2019).

28

[48]

[49]

[50]

[51]

[52]

[53]

[54]

[55]

[56]

[57]

REFERENCES

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas
Daniilidis. Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human
Pose. 2017. arXiv: 1611.07828 [cs.CV].
Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli.
“3D human pose estimation in video with temporal convolutions and semisupervised training”. In: Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. “Faster R-CNN:
Towards Real-Time Object Detection with Region Proposal Networks”. In:
arXiv:1506.01497 [cs] (Jan. 2016). arXiv: 1506.01497. url: http://arxiv.
org/abs/1506.01497 (visited on 03/18/2021).
Istvan Sarandi, Timm Linder, Kai O. Arras, and Bastian Leibe. “Metric-Scale
Truncation-Robust Heatmaps for 3D Human Pose Estimation”. In: 2020 15th
IEEE International Conference on Automatic Face and Gesture Recognition(FG 2020) (Nov. 2020). doi: 10.1109/fg47880.2020.00108. url: http://dx.doi.org/10.1109/FG47880.2020.00108.
Nidhi Seethapathi, Shaofei Wang, Rachit Saluja, Gunnar Blohm, and Konrad
P. Kording. “Movement science needs different pose tracking algorithms”.
In: arXiv:1907.10226 [cs, q-bio] (July 2019). arXiv: 1907.10226. url: http://arxiv.org/abs/1907.10226 (visited on 03/15/2022).
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna
Vedantam, Devi Parikh, and Dhruv Batra. “Grad-CAM: Visual Explanations
From Deep Networks via Gradient-Based Localization”. In: 2017, pp. 618–
626. url: https://openaccess.thecvf.com/content_iccv_2017/html/
Selvaraju _ Grad - CAM _ Visual _ Explanations _ ICCV _ 2017 _ paper . html
(visited on 12/12/2022).
Ajay Seth, Jennifer L. Hicks, Thomas K. Uchida, Ayman Habib, Christopher L. Dembia, James J. Dunne, Carmichael F. Ong, Matthew S. DeMers,
Apoorva Rajagopal, Matthew Millard, Samuel R. Hamner, Edith M. Arnold,
Jennifer R. Yong, Shrinidhi K. Lakshmikanth, Michael A. Sherman, Joy P.
Ku, and Scott L. Delp. “OpenSim: Simulating musculoskeletal dynamics andneuromuscular control to study human and animal movement”. en. In: PLOS
Computational Biology 14.7 (July 2018). Publisher: Public Library of Science, e1006223. issn: 1553-7358. doi: 10 . 1371 / journal . pcbi . 1006223.
url: https://journals.plos.org/ploscompbiol/article?id=10.1371/
journal.pcbi.1006223 (visited on 10/27/2022).
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks
for Large-Scale Image Recognition. arXiv:1409.1556 [cs]. Apr. 2015. doi: 10.
48550/arXiv.1409.1556. url: http://arxiv.org/abs/1409.1556 (visitedon 12/07/2022).
Richard Taylor. “Interpretation of the correlation coefficient: a basic review”.
In: Journal of diagnostic medical sonography 6.1 (1990). Publisher: Sage Publications Sage CA: Thousand Oaks, CA, pp. 35–39.
Thomas K. Uchida and Ajay Seth. “Conclusion or Illusion: Quantifying Uncertainty in Inverse Analyses From Marker-Based Motion Capture due toErrors in Marker Registration and Model Scaling”. In: Frontiers in Bioengineering and Biotechnology 10 (2022). Publisher: Frontiers Media SA.

REFERENCES

[58]
[59]
[60]

[61]

[62]

[63]

[64]
[65]

29

Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. Scotts
Valley, CA: CreateSpace, 2009. isbn: 1441412697.
Vasilis Vryniotis, Philip Meier, Nicolas Hug, Francisco Massa, and vfdev-5.
torchvision. https://github.com/pytorch/vision. 2021.
Logan Wade, Laurie Needham, Polly McGuigan, and James Bilzon. “Applications and limitations of current markerless motion capture methods forclinical gait biomechanics”. en. In: PeerJ 10 (Feb. 2022). Publisher: PeerJ
Inc., e12995. issn: 2167-8359. doi: 10 . 7717 / peerj . 12995. url: https ://peerj.com/articles/12995 (visited on 03/15/2022).
Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. YOLOv7:
Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv:2207.02696 [cs]. July 2022. doi: 10.48550/arXiv.2207.02696.
url: http://arxiv.org/abs/2207.02696 (visited on 12/12/2022).
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. 2017. arXiv:
1611.05431 [cs.CV].
Gary T. Yamaguchi and Felix E. Zajac. “A planar model of the knee joint tocharacterize the knee extensor mechanism”. In: Journal of Biomechanics 22.1
(1989), pp. 1–10. issn: 0021-9290. doi: https://doi.org/10.1016/00219290(89 ) 90179 - 6. url: https : / / www . sciencedirect . com / science /article/pii/0021929089901796.
Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola. “Dive into
deep learning”. In: arXiv preprint arXiv:2106.11342 (2021).
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the Continuity of Rotation Representations in Neural Networks. 2020. arXiv: 1812.
07035 [cs.LG].