—Objective: To automate identification of postural point -features from colour videos of children with neuro motor disability , during clinical assessment. The automatic identification of 13 points of interest (2, 6, 2, 3 points on the head, trunk, pelvis , arm respectively) is required to estimate the location and orientation of head, trunk, and arm segments , from videos of the clinical test “Segmental Assessment of Trunk Control” (SATCo) which is a test of seated postural control . Methods: Three expert operators manually annotated 13 point -features in every fourth image of 177 short (5 -10 second) videos (25 Hz) showing 12 children with cerebral palsy (ages: 4.52 ±2.4, male: 9), participating in SATCo testing. Linear interpolation for the remaining images resulted in 30,825 annotated images. Mean - pooling and max -pooling c onvolutional neural networks were trained with cross -validation, giving held -out test results fo r all children. Results: The point -features were estimated with error 4.40 ±3.75 pixels (mean -pooling ), and 4.49 ±4.45 pixels (max- pooling ), at approximately 100 images per second. Trunk segment angles (head, neck, 6 thoraco -lumbar -pelvic segments) were estimated with error 6.4° ±2.8° allowing accurate classification (F 1 > 80%) of deviation from a re ference posture at thresholds up to 3°, 3°, 2° respectively. Contact between arm point features (elbow, wrist) and supporting surface was classified at F 1 = 80.5%. Conclusion and Significance: This study demonstrates, for the first time, a technical solution to automate identification of i) a sitting segmental -posture including individual trunk segments , ii) changes away from that posture , and iii) support fro m the upper limb, required for the clinical SATCo. Index Terms —action recognition, artificial neural networks, mean -pooling, cerebral palsy, deep learning, feature tracking, human pose recognition, max pooling, pose estimation, pose tracking, SATCo, video analysis. I. INTRODUCTION primary aim of physical therapy for children with neuromotor disability, such as cerebral palsy (CP), is to improve postural control in order to enhance both fine and gross motor skills [1] with postural control of the head and trunk being of primary importance [2]. Accurate assessment of a child’s head and trunk control in sitting is thus essential . Manuscript received xxxxx; revised xxxxx; accepted xxxx. Date of publication xxxxx; date of current version xxxx. R. J. Cunningham, M. B. Sánchez , P. B. Butler , M. J. Southgate , and I. D. Loram are with Mancheste r Current clinical physical therapy assessments of controlled sitting ability for children with CP, although reliable, infer motor control status from the subjective observation of functional abilities [3]–[6]. The vital clinical need is for an objective measure of seated postural control [7]. A further limitation of these assessments is that they do not consider i) the multi -segmental nature of the trunk , ii) any comp ensatory use of the hands and arms to help maintain an upright posture in the presence of poor head/trunk control or , iii) give a precise definition of the correct posture . In contrast, the Segmental Assessment of Trunk Control (SATCo) [8] addresses these issues, providing a comprehensive assessment of seated postural control. It systematically assesses control of the neutral vertical posture in sitting at six discrete head/tr unk segmental levels and free sitting. The classification of neuromuscular control at a given head/trunk segment requires two conditions to be met : i) ‘alignment ’: the segment should remain aligned to the neutral vertical posture within a defined threshold and ii) ‘contact ’: there should be no contact between the upper limbs and the head/trunk or external surface [8]. Previous work [7], [9] has demonstrated th at the two components of ‘alignment ’ and ‘ contact ’ can be accurately classified from a clinical assessment and quantified using video recordings. However, t he methods used previously were semi - automated requiring manual initialisation of postural point features and necessitating significant human intervention to compensate for tracking drift and occlusion [7], [9] . The purpose of this study is to test whether this application is solvable using neural network methods, known as “Deep Learning” . Our objective is to automate identification of postural point -features from colour videos of children with cerebral palsy , during the crow ded environm ent of a clinical assessment in which the child interacts physically with one or more therapists . The identification of 13 poi nts of interest (2, 6, 2, and 3 points on the head, trunk, pelvis , and arm respectively), is required to estimate th e location and orientation of ten head, trunk, and arm segments to automate video -based analysis of seated postural control during the clini cal test SATCo. Metropolitan University, M1 5GD, UK. The authors express sincere thanks to the children whose data was used in this study, their families and to the staff of The Movement Centre, Oswestry, UK. Fully automated image -based estimation of postural point -features in children with cerebral palsy using deep learning Ryan Cunningham , Member , IEEE , María B. Sánchez , Penelope B. Butler , Matthew J. Southgate, and Ian D. Loram, Member, IEEE A > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 2 A. Overview and Justification of the Method Recent developments in machine learning [10]–[20] justify the value in exploring neural network methods for this application. T he goal i n image classification is to detect the presence of an object or feature and to achieve invariance with respect to the labels such that the classification in the final layer of the neural network should remain static in the presence of linear transformation (scale, translation, rotati on) of the input image. This can be addressed in neural networks, through the use of convolutional layers, which give rise to equivariance in the activations of the neurons, and through the use of max - pooling layers, which achieve local patch (typically 2x 2 patch) invariance to translation [21]. The use of multiple layers of convolution and max -pooling (deep learning) radically improves the neural network’s robustness to linear transformation of th e input space. However, following this method much of the precise information about the position of an object of interest within an image is lost [22], [ 23]. To retain the precise detection and tracking of point -features , [14], [18], [24], [25] one solution is to utilize deconvolutional layers and max-un-pooling layers [26], [27] . These layers, through learning, reverse the process of convolution, and recover the position information that was lost during max -pooling, to reconstruct a precise pathway to the full resolution image [13], [16], [17], [28] –[32]. However, the training time and runtime is relatively expensive . Another possible solution is t he use of patch -based convolutional neural network methods [11], but this method has been reported to have problems with fine precision [15]. There are hardwar e/software an d open framework solutions such as Kinect V1-V2, or OpenPose [33], which provide postural estimation. However, they do not consider the trunk a multi -segme nt structure, which is essential for the automation of SATCo. Also, existing open frameworks are not validated on children with neur omotor disability , which is essential for this clinical application. More recent sophisticated approaches are based on the c oncept of adversarial learning [34], but since their relevance is to a mor e general and complex problem set, they were judged inappropriate as an introductory baseline approach . Max-pooling is highly cited and is recognised as being the main problem to overcome in tasks relating to spatial feature recovery : thus a simple strategy might be to apply striding convolutions without pooling, to mimic the same down - sampling effect [35], while retaining high -resolution spatial feature information in the structure of the network. However, pooling is simultaneously recognised as one of the main strengt hs of the contemporary deep learning methods. We thus argue that , depending on the domain, a more efficient approach might be to change the pooling strategy from max -pooling to mean -pooling [36]. Efficient and effective spatial dimensionality reduction is achieved by taking the average of small patches of convolution maps, rather than the max : this retains the possibility, through learning, of convey ing precise position information between layers of abstraction via interpolation between adjacent patches, since the pooling function will vary with spatial transformations. Inevitably , the more complicated the task (estimating pose in arbitrary images from arbitrary viewpoint), the less effective this approach may be. However, s ince our problem is highly constrained (fixed camer a showing side profile of child) this approach is worth investigatin g. Mean-pooling provides a comparatively simple architectur e and therefore higher runtime efficiency, thus enabling feasible online (real -time) analysis , even without expensive hardware . Our approach was thus an appropriate deep convolutional architecture based on empirical pre -testing, which utili sed mean -pooling after every convolutional layer, and trained the network with no pre -training or transfer learning , to predict the points of interest of the child’s head and trunk directly from individual raw ima ges. For comparison, we trained an equivalent max -pooling network. II. METHODS A. Ethical Protocol Ethical approval for the study was obtained from the NHS Health Research Authority (NRES Committee South Central, United Kingdom) and from the University Ethics Committee. The study was conducted in accordance with the Declaration of Helsinki guidelines. B. Participants , Procedures and Measurements Data were collected in a previous study [7] from 12 children (9 males, 3 females, mean age 4.52 years ±2. 4, mean height 0.97m ±0.1, and weight 16.15kg ±7.5). All children had a diagnosis of CP and attended a specialist physical therapy centre. The number of sessions recorded per participant varied in relation to their routine attendance during the time the pr oject was running. The inclusion criterion was poor seated head/trunk postural control. The e xclusion criteria were fixed bony deformity or other structural problem of the spinal joints and if Figure 1. Point -feature and segment definitions. Squares show markers placed on the back of the participant: spinous process of the seventh cervical vertebra (C7), third, seventh and eleventh thoracic vertebrae (T3, T7 and T11), third lumbar vertebra (L3) and first sacral vertebra (S1). Dots show marker s located on the side and front of the child: right ear tragus, right temporal fossa (in a vertical line from the ear tragus when the head was in neutral position), greater trochanter and right anterior superior iliac spine (ASIS). Markers define trunk seg ments: head, neck, upper-thoracic (UT), mid-thoracic (MT), lower - thoracic (LT), upper-lumbar (UL), lower -lumbar (LL) and free sitting (Pelvis). Crosses show the landmarks tracked for the right arm (shoulder, elbow, wrist) , defining segments upper arm (UA) and forearm (FA) . > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 3 neither parent/guardian had sufficient understanding of written or spoken English to give informed consent. Informed written consent was obtained on behalf of their child from a parent or guardian, with written child’s assent where possible. Children wore shorts as was usual for their clinical assessments with girls w earing a crop top if required : this allowed accurate palpation of anatomical landmarks for marker placement . The SATCo was conducted according to the published guidelines [8], assessing control at six trunk segmental levels (head, upper -thoracic (UT), mid -thoracic (MT), lower -thoracic (LT), upper -lumbar (UL), lower -lumbar (LL) ), and Free Sitting, or as far as each child’s neuromuscular control permitted, within a single session [7]. Concurrent video was recorded at 25Hz from a JVC, HD Everio RX110 video camera mounted on a levelled tripod on the right side of the chi ld at a distance of 3.0m and a height of 0.70m. This view allowed recording of sagittal plane movements of the head and trunk and arm . Markers (small 2 cm3 coloured bock s) were placed on specific landmarks of the head, trunk and pelvis, to improve the late ral visualisation and tracking of the back landmarks following the model previously developed (figure 1) [7]. C. Generation of Point -Feature Labels A total of 34 sessions were recorded resulting in 17 7 videos , each of which was processed independently. Three operators , with expertise in labelling, used our custom created user interface in Matlab to manually annotate 13 point -features from 2 markers on the head, 6 markers on the trunk, 2 markers on the pelvis , and one marker on each of the right shoulder, right elbow and right wrist (figure 1) . The right arm was the only arm fully visible. This process resulted in 30,825 images, each with an associated set of 13, 2D coordinates describing the points of interest. Where a point -feature w as occluded, anatomical landmarks and visible markers, as well as adjace nt frames were used to infer the position of the occluded point -features. The custom user interface superimposed the previous (4 frames earlier) annotation, and operators updated the new position of the point -features by clicking nearby. This feature enabl ed rapid annotation, particularly for sequences or individual point - features which exhibited very little movement . In all sequences, every fourth image was annotated : linear interpolation was used to generate labels for the images without annotations at th e completion of manual annotation of the full sequence . D. Image Processing Prior to training, all images were cropped to a static region of interest around the children, then resampled using bilinear interpolation to 256x256x3 pixels. As this data set was relatively small , some additional processes were added during the training of the networks. To accommodate natural sagittal shift of the children relative to the bench and camera, the input images were randomly translated, sampling horizontal and vertical shift parameters from a real uniform distribution (single precision) in the range of [ -16 16] (pixels): this was repeated for each learning iteration (a forward pass, an error calc ulation, a backward pass, and a weight/parameter update) . The labels (point -feature coordinates) were augmented with the sampled translation parameters before each learning step. Bilinear interpolation was used to sample the pixel intensities, during trans formation. Conversion of pixels to metric space was not possible with these data, thus, labels were delivered in the pixel space. Following translation, local contrast normalization was applied via the GPU (runtime < 0.001s) with a local neighbourhood of 1 9x19 (pixels): this account ed for natural variation in contrast, brightness, and skin tone. These two processes , respectively , artificially increase d the size of the training data set, and increase d speed of convergence : both improving processes also impro ved generalization to the testing data set by limiting over -fitting. E. Network Architecture The network architecture selected was a moderately deep, 5 convolutional and mean -pooling layers , consisting of [64, 64, 128, 128, 256] (from high spatial resolution to low sp atial resolution) exponential linear unit s (ELU) , 3 nonlinear fully connected layers of 512 ELU units, and a final linear regression layer of 26 units (13x + 13y). The first convolutional layer used strides of 2x2, while all other convolutional layers used strides of 1x1. The spatial filter sizes in each convolutional layer were [9x9x3, 5x5, 5x5, 3x3 , 3x3] (from high spatial resolution to low spatial resolution). All pooling layers operated on neighbourhoods of 2x2, down -sampling convolution maps by a factor of 2. After 5 layers of convolution and pooling, the image representation layer had a spatial resolution of only 4x4 pixels with 256 nonlinear features. Down -sampling was not taken any further in order that some spatial detail was retained (figure 2). Each n etwork was trained on approximately 28,000 training images for approximately 500,000 learnin g iterations (18 cycles through the training set). Two different networks were trained in parallel on different GPUs. All networks were initialized according to the following scheme, based on the literature and our experience of training neural networks. L inear unit weights were drawn from a real Figure 2. Neural network architecture. Layers of convolutional Exponential Linear Unit (ELU) filters (green) are shown with spatial down -sampling (average -pooling) to learn a low -resolution spatial feature representation of the 256x256x3 (RGB pixels), local contrast normalized (LCN) (blue), in put image* (far left). Multiple fully connected ELU layers (yellow) then map that feature representation to the prediction of 13 points of interest, with a final linear regression layer (red). *actual full -colour image cannot be shown for ethical reasons; a cartoon/edge image is shown to illustrate the process . > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 4 (single precision) uniform distribution in the range [−√3 𝑓𝑎𝑛 𝑖𝑛+√3 𝑓𝑎𝑛 𝑖𝑛]. ELU unit weights were drawn from a real (single precision) uniform distribution in the range [−√6 𝑓𝑎𝑛 𝑖𝑛+√6 𝑓𝑎𝑛 𝑖𝑛], where the fa n in is the total number of the local (spatial) and feature inputs to any given unit in a convolutional layer, and the total number of feature inputs to any given unit in a fully connected layer. F. Deep Learning Software All neural network software was devel oped entirely within the group at Manchester Metropolitan University, with all code/software written solely by the first author using C/C++ and CUDA -C (Nvidia Corporation, Santa Clara, California). Only the standard CUDA libraries (runtime version 8.0 cuda.h, cuda_runtime.h, curand.h, curand_kernel.h, cuda_occupancy.h, and device_functions.h) and the C++ 11 standard library were used. All 12 neural networks were trained on an AMD Athlon X4 860K Quad -Core 3.7GHz CPU, 32GB (2400 MHz), with two Nvidia GTX 1080 GPUs. G. Training and Cross -Validation Adaptive moment estimation (ADAM) was used with default β1 = 0.9 and β 2 = 0.999 parameters, but with a smaller ( α = 0.0001) than suggested α = 0.002 parameter (learning rate) to account for non -batch (batch size of 1) learning. All parameters were empirically selected using a subset of the data, to check for quick (with respe ct to weight updates) and stable (no ‘exploding gradients’) convergence. Cross validation was executed with 12 folds, where for each set of network parameters and properties (architecture/units) 12 identical networks were separately trained using 11 of the 12 participants’ images and labels to train each network, and the remaining participant’s images and labels to test each network (split over the trials 50% validation, 50% testing). Participant’s images and labels used to test a ny of the 12 total network s were not used to test any of the other 11 networks. This process yielded genuine held -out test results for all 12 participants. Mean absolute error (MAE) was minimise d through online learning, which was interrupted every quarter pass through the training set (approximately 7000 learning iterations) , to record MAE test results from the 2 test sets, individually. If the MAE for either test set was lower than any previous recorded MAE for that test set, the network was saved to long term storage. When neither test set recorded a lower MSE for 32 consecutive test iterations, training was terminated. At the end of training, the network associated with the lowest loss for test set 1 was loaded to acquire results for test set 2, and vice versa , yielding true held out optimal results for both test sets. H. Classification of Head/trunk Segment ‘Alignment ’ After network training and production of held-out test output for all 12 children, all poin t-feature data were used to calculate the segment angles of th e head, neck and trunk segments as required for SATCo [7]. The SATCo classifies a segment with absolute dev iation more than 2 0° from the reference aligned posture as ‘misaligned’ [8]. Here, for this test of feasibility, the reference posture was defined per child as the mean posture over all levels and sessions , as defined by the labels, not by the neural network. For e ach of eight head -trunk segments, the label -derived angles and neural network -derived angles were thresholded at a range of angles (±1 to ±40 degrees) and classified as positive when above the threshold. Agreement between label, and network -derived classif ication was measured using accuracy, true positive rate, false positive rate, precision, recall, and the F 1-score ( table 2) . Since the class populations are severely unbalanced, precision (true positive/total predicted positive) , recall (true positive/total actual positive) and specially F 1 (harmonic mean of precision and recall) provide the most valuable measures. I. Classification of Upper Limb ’Contact ’ A fixed threshold was defined which, for any given image, was the position of the uppe r pelvis marker (Anterior Superior Iliac Spine (ASIS)) as defined by the labels, and not by the neural network. Then, for both neural network -derived point - features and for label -derived point -features, the position of the elbow marker was positively class ified as in contact with external support if it was to the left of the ASIS marker, and the position of the wrist was classified as in contact with external support if it was below the ASIS marker . III. RESULTS A. Image -Relative Point -Feature Localisation The labels and the neural network output comprise 26 real - valued numbers defining x and y coordinates of the 13 point - features. The neural networks were trained to minimise the absolute difference between their output and the labels. Results, Table 1. Neural network comparison to benchmark. Results are presented in the form of mean absolute error (MAE) of the difference between predicted (neural network) or tracked (benchmark) segment angles and manually annotated segment angles. The bold in the Dartfish column denotes the best results over all 3 methods. The bold in the neural network columns denote the best result over both neural network methods. The table shows that the neural network gave comparable performance with the benchmark, mean -pooling was slightly better than max -pooling, and fo r the MT segment both neural networks outperformed the benchmark significantly. Benchmark results do not include any occluded features, whereas the neural network results do. The benchmark was not capable of reliably tracking the arm segment. Dartfish Neural Network Sanchez, et a l [7] Mean -Pooli ng Max -Pooling Upper Arm - 6.7 ±9.5 7.5 ±10.6 Forearm - 9.9 ±12 10.7 ±12.1 Head 4.5 ±7 9.6 ±8.9 8.7 ±8.3 Neck 2.3 ±3.3 4.8 ±6.1 5 ±5.9 UT 5.3 ±9.8 6.2 ±5.8 6.2 ±6 MT 4.3 ±11.3 4.2 ±3.3 4.1 ±3.2 LT 3.2 ±4.5 5.1 ±4.1 5 ±4.1 UL 3.8 ±6.3 4.4 ±3.4 4.8 ±3.6 LL 7 ±10.2 8.7 ±7.4 8.9 ±7 Pelvis 3.7 ±7.9 8.5 ±7.4 8.6 ±7.3 Arm* - 8.3 ±8.9 9.1 ±9.4 Trunk** 3.7 ±5.1 5.6 ±2.6 5.7 ±2.4 Reference*** 3.8 ±5.2 6.4 ±2.8 6.4 ±2.7 All Segments**** 3.8 ±5.2 6.8 ±3.1 6.9 ±3.1 *Combined results of Upper Arm and Forearm **Combined results of UT, MT, LT, UL, LL ***Combined results of Head, Neck, UT, MT, LT, UL, LL ****Combined results of all segments > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 5 in terms of difference in pixels, showed that overall the neural network was able to locate each point -feature to within 4.40 pixels ±3.75 (figure 3). Image resolution was 256x256 pixels. B. Segment Angles To generate results comparable to the existing literature, segment angles were computed according to [7], [9] . Results for the relevant segments (head, neck, UT, MT, LT, UL, LL, pelvis) for our fully automated method recorded comparable performance, 6.4° (MAE ), to the current benchmark semi - automated method, 3.8° (MAE ) (table 1). Both neural networks compared very well with the benchmark, particularly in the trunk segments ( LL-UT), where the optimal MT error was actually given by the neural network methods and not Dartfish. The segments most contributing to the error were the head, upper arm, forearm , lower lumbar and pelvis, with errors between 8.5° and 9.9° (MAE ), whereas the trunk and neck segments gave errors between 4.2° and 6.2° (RMSE). Within a) b) c) d) Figure 3. Neural network visual comparison with labels. Showing representative held -out test results of neural network performance compared with the manual labels. The images* show sagittal views from one image in a video sequence of child sat on a bench in the centre of the image , and one or more physiotherap ists performing the SATCo test at a variety of segmental levels. Coloured circles mark the locations of the 13 point -features. There are 4 pairs of images, showing the labels (left image), and the neural network prediction (right image) from the same raw c olour image. All 4 examples illustrate high prediction accuracy in the presence of occlusion, particularly trunk (a), lower -thoracic and upper/ lower -lumbar trunk ( b), mid-thoracic, upper/ lower -lumbar trunk ( c), mid -thoracic, lower - thoracic trunk (c), upper/lower -lumbar trunk . *the actual colour images used in the analysis cannot be shown for ethical reasons, therefore cartoon images are shown to ill ustrate. Rocking Back and Forth Reaching Out Dropping Arms and Curving Spine Banging Drum Figure 4. Illustrative neural network motion tracking results. This figure shows the stability of the neural network predictions over a set of selected actions for 4 representative participants. Each of the 4 panels shows 3 images*, the leftmost image is t he start of the action sequence, while the rightmost image is the end of the action sequence ( 30 frames after the start), and the centre image is the middle of the sequence (1 5 frames after the start and 1 5 before the end). The coloured dots represent the point -features predicted by the neural network at the current frame, while the trailing coloured lines represent the historical pat h of the point - features predicted by the neural network in the previous 1 0 frames (0. 4 seconds). While there are some discrep ancies, it appears that each action sequence has been captured by the neural network. *the actual colour images used in the analysis cannot be shown for ethical reasons, therefore cartoon images are shown to ill ustrate. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 6 Table 3. Neural network classification of external support via the arm. Results are presented on the accuracy of the neural network in predicting contact of the arm with a supporting structure (the child’s own body, somebody else, the bench, or some other object). Classification rates in the form of accuracy, recall, precisio n and (or F 1-score) are very high. The elbow point -feature gives the highest true positive and lowest false negative rate of classification, while the wrist point -feature gives the highest precision. False negative rate in both sets of analysis are the mai n source of error. Point -Feature(s) #Positive Classes #Negative Classes Accuracy True Positive Rate False Positive Rate True Negative Rate False Negative Rate Precision F1-score Elbow and Wrist 5583 25242 92.1% 69.2% 2.9% 97.1% 30.8% 84.2% 76.0% Elbow 4458 26367 94.5% 77.9% 2.7% 97.3% 22.1% 83.2% 80.5% Wrist 1445 29380 96.0% 24.2% 0.5% 99.5% 75.8% 71.4% 36.1% Table 2. Neural network -predicted segment angle threshold analysis results. Results are presented in various forms of accuracy measure, on classification of segment alignment based on neural network and label derived segment angles. A fixed threshold of ±20 ° was used to classify the labels. Low numbers of positiv e classes lead to poor estimates of performance. In the MT, UT, neck and head segments, ample positive classifications provide good estimates of performance . False negative rate s are the main source of error. Segment Label Threshold Neural Network Thresho ld #Positive Classes #Negative Classes Accuracy True Positive Rate (Recall) False Positive Rate True Negative Rate False Negative Rate Precision F1-score Head 20o 20o 5003 25822 84.0% 48.7% 9.2% 90.8% 51.3% 50.7% 49.7% Neck 20o 20o 2074 28751 95.3% 61.4% 2.3% 97.7% 38.6% 66.3% 63.7% UT 20o 20o 959 29866 96.2% 23.3% 1.5% 98.5% 76.7% 33.7% 27.5% MT 20o 20o 296 30529 99.2% 39.2% 0.2% 99.8% 60.8% 65.5% 49.0% LT 20o 20o 208 30617 99.3% 7.2% 0.1% 99.9% 92.8% 39.5% 12.2% UL 20o 20o 147 30678 99.5% 2.7% 0.0% 100.0% 97.3% 100.0% 5.3% LL 20o 20o 909 29916 96.6% 0.1% 0.4% 99.6% 99.9% 0.8% 0.2% Pelvis 20o 20o 1102 29723 95.6% 3.4% 1.0% 99.0% 96.6% 11.8% 5.3% Figure 6. 40 randomly selected example neural network point -feature predictions. This figure shows images* and neural network predictions for a range of possible poses, randomly selected from all 30,825 images. Each of the 40 panels show an image of a child, and the neural netw ork prediction grouped by segment (arm = red line, head = green line, trunk = blue line, hip = cyan line). This figure is representative of the neural network’s ability to capture the pose of a child it has not been trained on. Ground truth is not shown to aid clear inspection, but predictions which do not match the image entirely are evident (e.g. the arm in row 5, column 2, and the head in row 5, column 7). In general, the neural network appears robust to new data and new poses. *the actual colour images used in the analysis cannot be shown for ethical reasons, therefore cartoon images are shown to illustrate. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 7 the trunk, the mid -trunk segments recorded strongest performance (UL, LT, MT). Comparison between neural network methods, mean -pooling and max -pooling, revealed marginal stronger performance for the mean -pooling method, with errors over all segments of 6.8 ° compared with 6.9 °. Notable differences were in arm and head segments. C. Classification of ‘Alignment’ For a fixed threshold of 20°, accuracy for segments ranged from 84.0 t o 99. 6% (table 2) . The number of positive classifications in the labels (‘misaligned’) ranged from 147 (UL) to 5003 (head). Accounting for the high imbalance in positive and negative classes, precision and recall measures were computed for UL, MT, UT, neck and head segments . The other segments did not contain enough cases to representatively summarise performance. Precision ranged from 0.8% to 100.0%. Recall ranged from 2.7% to 61.4%. False positive rates were low, ranging from 0. 0% to 9.2%. False negatives were the main source of error, with values ranging from 38.6% (Neck ) to 99.9% (LL). F1 ranged from 0.2 % (U T) to 63.7% (H ead). For all segments, F1 was very high for low thresholds and decreased as threshold increased ( figure 5). For the six trunk segments (UT, MT, LT, UL, LL, pelvis) neural network prediction was reliable (F 1 > 0.5) for thresholds up to 4 -5 degrees. For the neck and head, prediction was reliable F 1 > 0.5) for thresholds up to 1 4o and 12o respectively . D. Classification of Upper Limb ‘Contact’ From the labels, the ASIS marker defined dynamic lateral and vertical thresholds linked to the pelvis, with which to det ect contact of the arm with the trunk or support surface . This analysis revealed very high precision and recall rates of 85.7% and 76.0% respectively, with an accuracy of 9 3.3% (table 3) . Analysis of the individual point -features of elbow and wrist revealed the elbow position to be the most reliable point -feature in determining contact with the body (table 3) . The elbow recorded accuracy at 9 6.3% with 7 6.0% true positive classification and only 11.2% false negative classification, while the wrist re corded 2 0.5% and 7 9.5% true positive and false negative, respectively. The F 1-score was significantly higher in the elbow than the wrist, at 8 7.3% compared with 30.6%. Figure 4 in supplementary material shows a representative example of this process. IV. DISCU SSION The purpose of this study was to test whether the estimation of 13 postural point -features from colour videos of children with cerebral palsy was feasible using neural network methods. Estimation should be robust during the crow ded clinical environment of a SATCo test . The method should detect ‘alignment’ i.e. deviation of each of eight head/trunk segments from a reference orientation and ‘contact ’ of the upper limb with the trunk or support surface , to the accuracy required to fully automate video -based SATCo analysis of seated postural control in children with neuromuscular disorders . Deep convolutional neural networks were applied to an existing video dataset of SATCo assessment s of children with cerebral palsy. This same data set had been used in previous research to develop a semi -automated, objective method for quantifying postural control [7]. That previous method reported problems of tracking drift and feature dropout (loss of feature during tracking), due to 3D motion in the 2D image plane, as well as occlusion of point -features [7], [9] . The present study introduced deep learning methods to overcome thes e problems and provide a fully automatic approach , which predicts the point -features directly from single images (i.e. no tracking, no initialisation). We compared our deep learning methods to the benchmark feature tracking method , Dartfish [7], by measuring MAE between method -derived segment angles, and label -derived segment angles . The two neural network methods, max -pooling and mean -pooling, performed similarly, but in almost all segments the mean -pooling method gave superior performance. With these qualifications in mind, the benchmark method demon strated higher tracking accuracy than the neural network methods, in all segments except the mid -thoracic trunk segment (table 1) – an area in the trunk commonly occluded by physiotherapists in the scene . It is noteworthy that the benchmark method was expe cted to give the strongest performance because every time the feature points drifted or dropped out, the operator recovered the point manually – therefore it was certain to be close to manual labels. The upper limb segments were not addressed by the benchm ark method. However , the benchmark method requires initialisation and manually supervised tracking of every point -feature in the head, trunk and pelvis, whereas the proposed method locates all point -features, including an additional 3 point -features on the right arm directly from the raw image, with no user interaction. Machine learning methods require data to learn and this data set is very small, with only 12 children, of w hom only 3 are female . It is expected that additional data would improve substantially the performance of the neural network trained in this study. Additionally , encouraged by the results of this study , exploration of many available suitable methods [10]–[20] offers the prospect of improving performance independently of dataset size. To translate the accuracy of this neural network method into measures relevant to SATCo testing we assessed ability to classify ‘alignment’ and ‘contact’ as the two proces ses required Figure 5. Variation of F 1 with Threshold. For all segments (except the arms), F1 shows the agreement between neural network -derived and label -derived classification of ‘alignment’ for a range of thresholds ±1° to ±40°. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 8 by the SATCo test to determine , for each head/trunk segment tested, whether neuromuscular control is demonstrated . ‘Alignment’ : The angular deviation of a segment from a reference orientation, greater than a threshold (e.g. ±20 ° [8]) is classed as positive i.e. ‘misaligned’. During clinical SATCo the reference orientation is alignment to the neutral vertical posture. This study is testing feasibility to estimate point features and not testing feasibility of training a neural network to identify the neutral aligned vertical posture. Thus, we used the mean posture per child from the annotated labels as the reference orientation for both the label and network predicted alignment. Calculat ion of segment angles from point -features was fully automatic. Agreement between labels and predicted alignment for head and neck segments was mainly correct (F1 > 50%) at thresholds up to the values used by clinicians ±20 ° [8] and ±17 ° [7] (figure 3, supplementary material ). For the lower segme nts (UT, MT, LT, JL, LL, Pelvis) agreement between labels and predicted alignment was usually correct (F1 > 50%) at a lower range of thresholds up to 5o (figure 3, sup plementary material ). In clinical testing, a child may have ability to control a segment and yet not demonstrate that ability e.g. because the child is tired, cannot be bothered or does not want to play. Lack of demonstration of control proves little. However, if control is demonstrated, one demonstration is sufficient to establish that control is possible. For clinical testing, the error that should be minimised is to predict alignment when the child is misaligned (false negative ), and the co mplement t o be maximised is true classification of misalignment (true positive). The main contributor to disagreement between labels and neural network on the detection of misalignment was the false negative rate i.e. segments classified as aligned which are actuall y misaligned. The mean neural network false negative rate was approximately 76.7%, while the false positive rate was very low (average: 1.8%). The higher false negative rates lie in the lower segments and are related to the substantially fewer cases of mis alignment (#Positive Classes, table 2). Datasets containing more children lacking control of the lower trunk segments need to be collected. Results from this system can be visually verified by seeing if the predicted pose matches the image and video sequen ce. Generally visual inspection (figure 6 , and supplementary media ) shows the results are good . ‘Contact’: The threshold marker ( ASIS ) was defined only by the labels to provide for us, rather than the clinical user, the most accurate assessment of contact relative to a known external location. The results (table 3) showed that ‘contact’ could be detected with high accuracy, relative to the amount of data available , wit h a very high F1-score of 80.5% (figure 4, supplementary material ). The arm had no physical markers (small blocks) and, if present, these would have helped the neural network to locate the point -features. Nevertheless , the neural network was able to locate the elbow and the wrist in a given image in the absence of artificial visual clues, with sufficient accura cy to detect contact with an arbitrary body. Visual inspection (see supplementary videos) of some of the neural n etwork predictions indicated that where the left arm was in view above or below the right arm, the network became confused about the position of the wrist, predicting a point approximately in the average location between the left and right wrist. In princi ple this error would be addressed with more data giving more training examples to discriminate . In future work, we propose using an “RGB -D” camera . The addition of depth data to complement the colour images would remove the ambiguity about left and right arm in the 2D sagittal plane . The results of this study justify further development including the collect ion of additional data, and the application and tuning of more sophisticated and established techniques, such as the Mask R -CNN [20], or the adversarial PoseNet [34]. Our experience suggests that there is no necessity to replicate the same volume of labe ls per additional child in the data set. Performance would likely increase if additional variation in the labels and images were introduced in the form of additional children. The current study generated over 30,000 labels. The generation of 30,000 labels from more children would add useful variation in body shape and type, developmental age, and severity of disability , while retaining the intraparticipant variation in range of motion per segment. From these results, we estimate that between 30 and 50 child ren, each with 1,000 annotations would result in a strong data set for delivery of a highly accurate neural network analysis. If the number of children was increased to around 100, with the number of labelled images around 100,000, this would allow for lar ger cross -validation testing batches, reducing the bias when regularizing by early stopping of training of the neural network . V. CONCLUSIONS This study demonstrates, for the first time, the technical ability to automate identification of i ) a posture comprising individual segments ii) changes away from that posture and iii) support from the upper limb to a level of accuracy and sensitivity comparable to the current SATCo clinical standard [8] in children with cerebral palsy . Application of our method can be widened to include other participant groups, such as typical ly developing infants and children with neuromuscular disease . Our method applied a standard con volutional architecture, using mean -pooling to retain some ability for point -feature localisation. It has potential to be accessible and reproducible by other laboratories worldwide through the use of publicly available code repositories. We have demonstra ted high levels of performance (F1 > 50%) , through state -of-the-art cross -validation, on a small data set of only 12 children: this proves efficacy and feasibility for expansion into larger sample sizes. We have also demonstrated that labelling data is straightforward. Our results justify the collection of additional data that is designed for purpose (i.e. no markers on the child), the investigation and comparison of a wider range of more sophisticated deep learning methods, and the introduction of depth in formation to complement the colour images. This will enable a fully automated assessment to address the vital clinical need for an objective measure of seated trunk control , to enhance planning and monitoring of a wide variety of interventions in children with neuromotor disability. > REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE -CLICK HERE TO EDIT) < 9