dataesr/scientific-paragraphs-categorization huggingface dataset:
This is thought to occur because highly expressed proteins need highly optimal sequences to avoid aggregation potentially driven by translation errors. However, most studies comparing abundance with sequence constraints have relied on a small number of species. In our analysis we found a significant correlation between protein abundance and predicted protein conservation in most of the 103 species in this dataset (Fig. 4C ). Protein abundance is normalised against length and expressed as the log2 fold change compared to median abundance in that species. Conservation is summarised for a protein as the mean number of predicted deleterious variants across positions. A similar correlation level is observed for Sequence UNET predictions, including PSSM prediction and frequency classification, and SIFT4G scores in H.
Discussion Variant effect prediction (VEP) is a central part of many computational genetic analyses, allowing researchers to assess new genomes or patient sequences, prioritise variants for follow up experiments and identify important functional and structural features of proteins. This has led to a range of VEP tools, using empirical and machine learning methods to predict deleteriousness from sequence and structure. However, most tools are restrictive to run, particularly for large scale analyses involving hundreds of proteins. They can be computationally expensive, often having slow multiple sequence alignments or more recently machine learning model training as a limiting step, and are often awkward to install correctly. Precomputed results are available for human proteins and some model organisms, for example in the MutFunc  23, 30  or dbNSFP  31, 32  databases and the webpages of many tools.
Despite this, it is still difficult for many scientists to generate predictions for the proteins they need, especially when analysing a large number of proteins or less studied organisms. In an attempt to fill this niche, we developed a versatile neural network VEP that is fast, lightweight and easy to use, able to rapidly make exhaustive predictions about many proteins on a regular laptop and scale to multi-proteome analyses on compute clusters. Neural networks provide a powerful approach to fast, high-quality predictions, utilising large datasets and a long, computationally intensive training process to distil complex relationships into numerical weights matrices. This means predictions can be made efficiently as long as the network architecture uses optimised operations and doesn't require expensive external computations, for example multiple sequence alignment or structure relaxation.
Similarly, GraphCNNs are performant and have been shown to perform on protein structure tasks  [36] [37] [38]  , and their position invariance makes them a natural approach to including structural features. We experimented with various other methods of including structure, including torsion angles and calculated feature profiles, but found GraphCNNs gave best performance and efficiency. The lack of labelled deleterious variant data at the scale required for deep learning led us to first capture general sequence and variant properties by training the model to predict variant frequencies, either as a PSSM predictor or a low frequency variant classifier, and then finetune for pathogenic variant classification using ClinVar data. This is similar to the protein language model paradigm, in which large models are first trained to predict amino acid sequences, capturing general properties, and the representation vectors they produce can be used as input into smaller downstream models.
However, the size and design of the model makes it much more computationally efficient than most language models. The base Sequence UNET model achieves state of the art performance at de novo PSSM prediction and frequency classification, both independently useful tasks, and the fine-tuned models reach top level performance at pathogenicity prediction, although the base model only generalises moderately well. There are also potential questions about performance on unstructured proteins, which are missing from ProteinNet, but this could be addressed by expanding the training data beyond structured proteins. Variant frequency can be measured analytically and other models can predict pathogenicity to a broadly similar accuracy, meaning the major strength of our model is computational efficiency.
There are various routes available that could improve our model in future, while maintaining computational efficiency. The role of protein structure is an obvious target for change, since it currently only adds a small performance boost despite structure being known to be critical for protein function. For example, a richer graph network section, graph attention mechanisms or pre-training the structural section to encode structural properties could all potentially improve performance. Removing the structure option altogether could also be beneficial because it would allow the model to be trained on a much larger sequence dataset, exposing it to more sequence variation and reducing the bias towards protein types with determined structures. Large sequence datasets, and lots of training data in general, has been found to greatly improve performance in many other models  7, 17, 39.
This would make the model setup more similar to protein language models, which also train to predict amino acid propensity on large sequence databases. The UNET sequence CNN portion of the network could also be adjusted, either by tweaking the current connections and parameters or switching to an alternate sequence processing architecture. For instance, powerful attention and transformer architectures could be incorporated into the network or used as a basis for a new model that maintains computational efficiency as a goal, although it does come with an inherent computation cost compared to convolution. We have demonstrated a highly efficient, performant model for variant frequency and effect prediction, which enables larger scale analyses than have previously been possible with VEP packages, demonstrated by our multi-proteome conservation analysis.
Most recently, several methods have been proposed on body reenactment from source images  [Chan et al. , 2019; Esser et al. , 2018; Liu et al. , 2019; Ma et al. , 2017; Mir et al. , 2020; Sun et al. , 2020]. The ultimate goal of these methods is to create a video where the body of a target person is reenacted according to the motion extracted from the monocular video. The motion is estimated considering the set of poses of a source person. Despite the impressive results for several input conditions, there are instances where most of these methods perform poorly. For instance, the works of  Chan et al. [2019]  and  Wang et al. [2018] , only perform good reenacting of the appearance/style from one actor to another if a strict setup has complied, e. g. , static backgrounds, a large set of motion data of the target person to train, and actors in the same distance from the camera  [Tewari et al.
In other words, they do not suffer from quality instability when applied in contexts slightly different from the original ones (a small difference in camera position, uncommon motions, pose translation, etc. ) and they can handle different morphologies in the retargeting. Moreover, we performed experiments using a newly collected dataset containing several types of motions and actors with different body shapes and heights. Our results show that a technique applying 3D representation of people can still exhibit a competitive quality compared to recent deep learning techniques in generic transferring tests. Our approach achieved better results compared with end-to-end 2D learning methodologies such as the works of  Wang et al. [2018]  and  Chan et al. [2019]  in most scenarios for appearance metrics as structural similarity (SSIM), learned perceptual similarity (LPIPS), mean squared error (MSE), and Fréchet Video Distance (FVD).
The main technical contributions of this work are as follows: i. A unified methodology carefully designed to transfer motion and appearance from video to video that preserves the main features of the human movement and retains the visual appearance of the target character, as shown in Figure  1. petitive visual retargeting quality compared to state-of-the-art neural rendering approaches, even without computationally intensive training; iii. A novel data-driven formulation for transfer appearance and reenact human actors that produces a fully 3D controllable human model, i. e. , the user can control the human pose and rendering parameters, as shown in Figure  1. 4; iv. A dataset comprising several videos with annotated motion restrictions. We demonstrate the effectiveness of our approach quantitatively and qualitatively using sequences from this dataset and publicly available video sequences.
, Coutinho, T. , Azevedo, R. , Martins, R. , and Nascimento, E. (2022). Creating and Reenacting Controllable 3D Humans with Differentiable Rendering. In WACV. The dataset and retargeting code are publicly available to the community at: https://www. verlab. dcc. ufmg. br/retargeting-motion and the presentations of our two methods are available at: https://youtu. be/seZfaiPoof4 and https: //youtu. be/BkS4AsiR1es.
Outline From here on, this dissertation is organized in the following chapters: Chapter 2 provides the reader an overview of the main and more recent techniques that are associated with our problem, i. e. , human motion estimation, motion transferring, and image synthesis. In Chapter 3, we describe the two proposed methods to transfer motion and appearance between videos. Following in Chapter 4, we present our carefully designed dataset to evaluate transferring human motion and appearance from video to video. In sequence, we present an analysis of our methodology quality in comparison to state-ofthe-art techniques in Chapter 5 and discuss the limitations of our methodology. Finally, in Chapter 6, we present our conclusions and highlight future research directions.
3D Human pose and shape estimation In general, motion transferring approaches from video to video contains a phase of human pose/shape estimation, i. e. , they estimate the configuration of the human body in a given image or a sequence of images. A simple approach is to employ a 2D representation of the human body pose in the image plane  [Chan et al. , 2019; Esser et al. , 2018; Wang et al. , 2018] , i. e. , understanding pose estimation as the problem of localizing anatomical keypoints or body joints in the image plane  [Cao et al. , 2017; Riza Alp Güler, 2018; Simon et al. , 2017; Wei et al. , 2016]. Methods of 2D human pose estimation have made significant progress during the last years, in a large extent due to the creation of large datasets  [Andriluka et al. , 2014; Lin et al. , 2014; Riza Alp Güler, 2018]  with annotated joint positions or dense correspondences from a 2D image to a 3D human shape.
, 2015]  or by regression or model fitting the 2D joints  [Akhter and Black, 2015; Martinez et al. , 2017]. Two-stage methods use priors to deal with pose ambiguity. Most of the priors are about the limb-length or proportions  [Barron-Romero and Kakadiaris, 2001; Parameswaran and Chellappa, 2004; Ramakrishna et al. , 2012]. In this context,  Akhter and Black [2015]  collected a dataset that includes a wide variety of stretching poses performed by trained athletes and gymnasts. Then they learned pose-dependent joint angle limits from the data and proposed a novel prior based on these joint angle limits. Two stage-methods benefit from being more robust to domain shift, but the major drawback of these methods is that their accuracy is bounded by the capacity of the 2D joints and the pose priors to explain the real poses.
There is still a lack of dataset of 3D poses for people in the wild since 3D data acquisition requires expensive setups. It is challenging to acquire data outside controlled laboratory conditions. However, after the introduction of Human3. 6M dataset  [Ionescu et al. , 2014b] , which contains 3. 6 million high-resolution images with annotated 2D and 3D joint locations, many modern methods were proposed to estimate 3D joints directly from images using deep learning frameworks  [Ionescu et al. , 2014a; Li and Chan, 2014; Mehta et al. , 2017; Zhou et al. , 2016b]. Since monocular reconstruction is inherently scale-ambiguous, the input image is commonly cropped to the bounding box of the subject before 3D pose estimation  [Ionescu et al. , 2014a]  and the output is a subject with normalized height.
Li and Chan [2014]  report that predicting positions relative to the parent joint of the skeleton improves the performance, but relative positions are a prohibitive simplification to motion transferring. Aware of the importance of global position and the fact that models trained only on laboratory images do not generalize well to the real world,  Mehta et al. [2017]  explore the use of transfer learning to leverage the highly relevant mid-and-high-level features learned on readily available in-the-wild 2D pose datasets in conjunction with the existing annotated 3D pose datasets. They reconstruct global 3D poses using a generalization of Procrustes analysis for projective alignment. Despite the importance of 3D joint locations to many applications in computer vision, 3D joint locations are sparse and do not constrain each joint's degrees of freedom, and do not ensure that limbs are symmetric and have the correct length.
This training is computationally intensive and takes several days on a single GPU. In order to overcome these limitations,  Liu et al. [2019]  proposed a 3D body mesh recovery module to disentangle the pose and shape; however, their performance significantly decreases when the source image comes from a different domain from their dataset, indicating that they are also affected by poor generalization to camera viewing changes. Recently,  Sun et al. [2020]  also proposed a data-driven method where projections of the reconstructed 3D human model are used to condition the GAN training, in order to maintain the structural integrity of the transfer to different poses. Nevertheless, all analyses were made in a strict setup where the person is standing parallel to the image plane, and the considered motions have reduced lateral translations.
Differentiable Rendering Most methods to estimate human body geometry and texture rely on supervised training regimes and costly annotations, which makes collecting the data challenging and expensive. Thus, there are recent efforts towards leveraging 2D information and differing levels of supervision for 3D scene understanding. One of the approaches is integrating graphical rendering processes into neural network pipelines. Differentiable renderers (DR) are operators allowing the gradients of 3D objects to be calculated and propagated through images while training neural networks. As stated in  Kato et al. [2020] , DR connects 2D and 3D processing methods and allows neural networks to optimize 3D entities while operating on 2D projections. Loper and Black [2014]  introduced an approximate differentiable render that generates derivatives from projected pixels to the 3D parameters.
). The 2D semantic body labels are computed using the  Gong et al. [2018] 's body parsing model, which we fine-tuned using the people-snapshot dataset  [Alldieck et al. , 2018]. After computing the semantic map of the body, for each contour point (red squares in Figure  3. 8) in the map, we define the vertex from the body mesh with the same semantic and the smallest Euclidean distance to the contour point as a control point (blue circles in Figure  3. 8). The Euclidean distance is computed between the contour point and the 2D projection of the vertex. Each control point will receive a target position given its correspondent contour point. These control points and their new locations guide the deformation of the mesh to fit the shape into the semantic map's contour. In the deformation, we seek a new body mesh Q that is a locally rigid transformation of the source body mesh P , following the control points given by the semantic contours.
Method II: 3D Differentiable Human Rendering Image-based rendering techniques like our previous technique are effective solutions to create 3D texture-mapped models of people, capable of synthesizing images from any arbitrary viewpoint without using a large number of images. On the other hand, image-based rendering methods cannot improve the visual quality of the synthesized images by using more data when available. Furthermore, the deformation process proposed in our previous method is not fast enough to be used in real-time applications. Thus, we offer a strategy to take advantage of all available data and, in addition, reduce inference time at the cost of increasing preprocessing time (training time). Therefore, we propose a data-driven method, i. e. , a method that requires training a different model for each target person to perform the transferring.
Chapter 4 Human Retargeting Dataset and Evaluation Datasets are one cornerstone of recent advances in Computer Vision. While there are many large datasets available for human shape and pose estimation  [Andriluka et al. , 2014; Lin et al. , 2014; Mahmood et al. , 2019] , existing motion retargeting datasets are yet rare, hampering progress on this area. Due to the lack of suitable benchmark datasets, recent works on neural view synthesis and body reenacting  [Chan et al. , 2019; Esser et al. , 2018; Wang et al. , 2018]  only analyze their results in qualitative terms or quantitative terms to self-transfer in which the source and target actors are the same subjects. The provided data is adapted to the requirements of their method setup, and it is hard to perform a comparison with other methods on this data.
Existing Datasets Existing video retargeting datasets are still rare. Villegas et al. [2018]  provided human joint poses from synthetic paired motions, however, paired visual information is not available, limiting the appearance transferring. Chan et al. [2019]  made available videos with random actor movements that can be used to learn the appearance of the target actor. However, the provided data is limited to their setup requirements, and it does not allow the analysis and comparison with other methods. Liu et al. [2019]  presented a set of videos with random actions of target subjects, as well as videos of the subjects performing an A-pose movement. This set enables methods focusing on modeling 3D human body estimation or using few keyframes to be executed using their data. On the flip side, the lack of paired motions limits motion and appearance retargeting results in quantitative terms, where the source and target actors are different subjects.
Human Retargeting Dataset To evaluate the retargeting and appearance transfer with different actor motions, consistent reconstructed 3D motions, and with human-to-object interactions, we created a new dataset with paired motion sequences from different characters and annotated motion retargeting constraints. For each video sequence, we provide a refined 3D actor reconstructed motion and the actor body shape estimated with  Alldieck et al. [2018]. The refined reconstructed 3D motions and 2D-3D annotation of interactions were collected by manual annotation. The provided motions are not prone to typical motion artifacts such as bended knees, body shape variations, and camera-to-actor translation changes. Figure  4. 1 shows the annotated labels. Our carefully designed dataset comprises eight subjects.
To keep the dataset with diversity, we choose participants (subjects S0 to S7) with different gender, sizes, clothing styles, and body shapes. Figure  4. 2 shows the subjects, their respective body models and labels. A short description of these subjects is as follows: • S0: A female character, 1. 65 meters height, and neat hourglass body shape; • S1: A male character, 1. 85 meters height, and inverted triangle body shape; • S2: A male character, 1. 70 meters height, and rectangle body shape; • S3: A male character, 1. 83 meters height, and pear body shape; • S4: A male character, 1. 83 meters height, and lean column body shape; • S5: A male character, 1. 81 meters height, and apple body shape; • S6: A male character, 1. 84 meters height, and rectangle body shape; • S7: A female character, 1.
e. , we evaluate the quality of one synthesized frame as a better answer between the frame and a small window of frames in the paired video. The window of acceptance w is estimated for each pair of videos according to: w(V 1 , V 2 ) = max(15, 2 × (|len(V 1 ) -len(V 2 )|), (4. 1) where len(V ) is the number of frames in the video V. Equation  4. 1 captures how much two videos are not synchronized allowing the synthesized frames to match with the paired video. The lower bound value of 15 frames was empirically selected by a visual analysis of our videos. Aside from the image quality metrics, we also propose to evaluate the approaches with fake image detectors. Since our dataset provides paired motion sequences, we can evaluate the retargeted frames' quality and realism with an image forgery detector.
Videos for Qualitative Analysis Although our dataset contains several videos, it is essential to evaluate the behavior of our methodology in less controlled data acquisition. Thus, we provide three videos acquired under uncontrolled conditions. In our experiments, we used these videos to analyze the effectiveness of our method with more complex motions, background, and varying illumination. A short description of these videos is as follows: • joao-pedro: Video with moderate frequency motions and a static background, where a 1. 80 meters height male character is walking and interacting with a cone in the floor. The motion constraints are in the feet, and the hand, where the person touches the top of the cone object; • tom-cruise 2  : Video with moderate frequency motions and large translation in all directions, where a 1.
Ablation Study To verify that the space-time motion transfer optimization, motion regularization, semantic-guided deformation, and visibility maps contribute to our approach's success in producing more realistic frames, we conducted several ablation analysis using the motion sequences from our dataset. Table  5. 1 shows the results of our ablation study in terms of MSE and FVD of five ablated versions of the Method I. We draw the following observations. First, the best result is achieved when the full method is applied. Second, removing the 2D/3D constraints reduces the performance in terms of MSE. These constraints play a key role in the compliance of the poses to motion constraints. By removing them, large fragments from the background are computed as part of the retarget character body when computing MSE using the paired video, leading to the worst MSE value.
Quantitative Analysis We performed the video retargeting in all sequences in our dataset, including several public dance videos also adopted in the works  Chan et al. [2019]  and  Liu et al. [2019]. Table  5. 5 shows the comparison of our approach in the dataset considering the motion types and pair of actors. We can see that despite not being dataset-specific, V-Unet and iPER did not perform well when the reference image is not from their datasets. One can see that our method outperforms, on average, all methods in considering SSIM, LPIPS, MSE, and FVD metrics. Regarding the experiments considering the pairs of actors, our first approach also achieved the best average results in all metrics. In particular,  In each sequence: the first row shows the worst generated frame for each method and the second row presents the best generated frame for each method.