dataesr/scientific-paragraphs-categorization huggingface dataset:
B. KINSHIP DATASETS To train kinship models, it is crucial to have an image dataset annotated with kinship relationships. Over the years, multiple datasets with different characteristics in terms of image quality, cropping, number of images, kin relationships, and metadata have been introduced. Lu et al. [31]  introduced the KinFaceW-I and KinFaceW-II datasets. The face images were captured under uncontrolled conditions with no constraints on pose, occlusion, or lighting. Robinson et al. [32]  presented the Families In the Wild (FIW) dataset. It consists of 1,000 families with over 10,000 family photographs and metadata on 11 types of kinship relationships. The FIW dataset was further expanded  [33]  to include additional data on underrepresented families, and increased the number of photos to 30,275.
Qin et al. [34]  proposed the TSKinFace database, which consists of families of 2589 individuals. Fang et al. [35]  presented the CornellKin dataset of 150 pairs of public figures and celebrities, along with images of their parents or children. The Family-101 dataset  [36]  was introduced in order to study the similarity of characteristics among family members. This dataset consists of 101 families with 607 individuals and 14,816 images. Xia et al. [37]  proposed the UB Kinface dataset to evaluate and analyze algorithms for the kinship verification task. The images are based on real collections of celebrities and politicians. The Siblings Database  [38]  consists of two main subsets: a high quality database (HQFaces) and a low quality database (LQFaces). The HQFaces consists of 184 images (92 siblings pairs) taken by a professional photographer with uniform background and controlled lighting.
The LQFaces contains 98 sibling pairs found on the internet, most of which are celebrities. A common aspect of the existing datasets is that the faces are either tightly cropped, of poor quality, or have an insufficient number of samples, which makes them difficult to use for the task of face synthesis. The Next of Kin dataset, presented in this paper, fills an obvious research need and features high-quality face images that enable the development and training of contemporary generative models, capable of producing photo-realistic high-resolution (artificial) face images.
C. KINSHIP RECOGNITION AND SYNTHESIS 1) KINSHIP RECOGNITION Most kinship-based research focused on the task of kinship recognition. One of the first methods for classifying parent-child pairs was presented in  [35] , which used lowlevel features, e. g. , average skin region color and gradient histogram. Xia et al. [39]  proposed a subspace learning method to obtain a more discriminative kinship verification problem. This work was extended in  [40]  using Gabor filters, metric learning, and subspace transfer learning to obtain improved results. Fang et al. [36]  presented a face reconstruction method by linear combination of database parts, which was used to determine the family membership of a particular person. Wang et al. [41]  proposed a denoising autoencoder based on metric learning for kinship recognition.
After the final child latent code is synthesized, it is passed through a decoder model. ChildNet uses StyleGAN2  [5] , which is capable of synthesizing high-resolution face images, as the decoder model. Following the practice of contemporary models that edit face images using StyleGAN models  [17] ,  [18] ,  [29] , we base the mapping model in the extended latent space W + , which encompasses multiple 512-dimensional latent vectors. The exact number of vectors depends on the model resolution. Since we use a model with 1024 × 1024 image resolution, the latent code space consists of 18 (concatenated) 512-dimensional vectors. The extended latent code can be divided into coarse, medium, and fine parts as these have been shown to influence different facial aspects of the synthesis process  [7].
2) LOSS FUNCTION The loss function contains three main terms. The identity loss term ensures that the synthesized child image and the ground truth child image look visually similar. We achieve this with the ArcFace recognition model  [45]  R by maximising the cosine similarity between the model embeddings: L id = 1 -cos(R(G(w C )), R(I C )), (5) where I C denotes the ground truth child image. Our next objective is to ensure ChildNet synthesizes diverse child appearances. This can be achieved efficiently in the latent space, since the StyleGAN2 training regime aims to map a fixed-size latent code step to a fixed-magnitude image change  [5]. Therefore, we propose a triplet loss term in the latent space. We denote an anchor (predicted) child latent code as w C , a positive code as w + C = E(I + C ) and a negative code as w - C = E(I - C ), where I + C denotes the ground truth child image and I - C denotes an image of a non-related child.
IV. NEXT OF KIN DATASET A. MOTIVATION As shown in the Related Work section, the existing datasets are limited both in terms of subjects and kinship relationships (e. g. , mother-son/daughter, father-son/daughter), while more recent datasets contain a larger number of families with kinship relationships extending beyond just two generations. However, as shown in Table  1  and Figure  5 , the existing datasets are still mainly designed for the study of automatic kinship recognition techniques and generally do not contain data that allow development and training of high-quality image synthesis models. We introduce the Next of Kin (NoK) dataset, which contains high-resolution face images suitable for the task of kinship face synthesis.
B. DATA COLLECTION We collected the dataset using the family relationship data from the FIW dataset. Following the common practice in the established in-the-wild datasets  [51] ,  [52] ,  [53] , we collected NoK images from the internet that were augmented with additional families not present in the FIW dataset. To ensure high image quality, we manually filtered the images, removing images with occluded faces and poor visual quality. We selected 512×512 as the optimal image resolution, and set the minimum acceptable resolution to 256 × 256. All images were rescaled to 512 × 512 using the CelebAHQ image  preprocessing steps  [6]. For the subjects with low image counts, we searched for additional high-resolution images.
C. DATA PROCESSING The image preprocessing steps are closely related to the ones used in the creation of the CelebAHQ dataset  [6]. The first step involved predicting facial landmarks and cropping the face image with the correct orientation. To achieve this, we used RetinaFace  [54]  as our primary model for predicting facial landmarks. All landmarks were visually inspected and manually corrected for any irregularities. Based on the corrected landmarks, a face image was rotated and cropped. Finally, we added a corresponding w ∈ W + latent code for each image by projecting each image onto the StyleGAN2 FFHQ model. We added information about gender and ethnicity 2  of each subject, as well as information about age and emotion for each image.
D. DATASET CHARACTERISTICS The NoK dataset consists of 3, 690 high-resolution images from 553 subjects that correspond to 161 families. The average number of images per subject is 6. 67 ± 4. 48 with a total of 127, 719 triplets. We provide official training, validation, and test splits, taking special care not to overlap images of the FIGURE 6. Next of Kin subject and image characteristics. We use DeepFace  [49] ,  [50] , a model for facial analysis, to analyze the dataset. The gender and ethnicity distributions are calculated on a per-person basis within the dataset, while the age and emotion distributions are calculated across all images in the dataset. same individual between splits. We analyzed the data using DeepFace, a face analysis model, to obtain per-person and per-image metadata in the form of gender, ethnicity, age, and emotion.
The distribution of the metadata statistics is shown in Figure  6. To assess the image quality of the NoK dataset compared to other kinship datasets, we perform an image quality assessment. The assessment is based on the state-of-the-art noreference image quality assessment model MANIQA  [48]  that estimates a quality score for each image. Since the model predicts scores for 224 × 224 images, the lower-resolution images are upsampled using bilinear interpolation, while higher resolution images are center-cropped to the target resolution. The model results are shown in the last column of Table  1. The NoK dataset has a higher average image quality score than other kinship datasets, and at the same time contains more images than other high-quality kinship datasets.
V. EXPERIMENTAL SETUP A. DATASETS AND EXPERIMENTAL SPLITS 1) KINSHIP MODELLING We compare ChildNet with existing methods using two datasets. The first dataset, FIW, has traditionally been used to train kinship face synthesis models. The second dataset is the proposed NoK dataset. We use the official training and validation splits for both datasets. The testing procedure for FIW is based on the official test triplet parent-child data, but we extend it by using up to five images per person and creating all possible combinations of triplet images to obtain more test data. Furthermore, we remove the images whose diagonal is less than 90 pixels long. In the end, our test data consists of 15, 815 triplet images for FIW and 12, 265 triplet images for NoK.
3) KINSHIP MODULE Each BaseNet of The Kinship Module consists of five-layered fully-connected networks. We use a dropout with a rate of p d = 0. 5, batch normalization, and Leaky ReLU as the activation function in all hidden layers. We train ChildNet on FIW and NoK datasets using the official training and validation splits. The batch size is set to 16. We use the Adam optimization algorithm  [57]  with a learning rate of η = 3 • 10 -4. The weighting factors are determined experimentally and are set to λ id = 1, λ tri = 10 -3 , and λ mag = 1. The triplet margin value is set to δ = 0. 1. We evaluate ChildNet against several competing models: DNANet  [1] , HeredityGAN  [2] , and StyleDNA  [3]. The DNANet and HeredityGAN models are implemented from scratch, whereas the StyleDNA model is based on the officially released code.
2) IDENTITY SIMILARITY, UNCONDITIONAL EXPERIMENT To assess the accuracy of the synthesized identity, we evaluate identity similarity using FaceNet  [59]  and ArcFace  [45]  face recognition models. These models are used to extract face embeddings from the synthesized and the real child images. We propose an evaluation strategy that does not use the negative child pairs due to the stochastic nature of sampling. Furthermore, we argue that the similarity of the synthesized child to the real child (positive pair) is more important than the similarity to a random child (negative pair). Therefore, our evaluation is based solely on the similarities of positive pairs, i. e. , similarities between the synthesized and real child images. Compared to previous kinship face synthesis approaches, our testing procedure also uses a much larger number of testing samples, consisting of 15, 815 triplets for the FIW dataset and 12, 265 triplets for the NoK dataset.
We evaluate the models by constructing a histogram of the predicted cosine similarities and a histogram of perfect cosine similarities (value 1 with 100 % confidence). We measure the discrepancy between these two histograms using the earth mover's distance. Additionally, we report the mean of the cosine similarities. The histogram results for the FIW and NoK databases are displayed in Figure  VI-A2 , showing that ChildNet produces higher similarity scores than the compared models. The quantitative results in Table  2  confirm this finding, with ChildNet achieving the best results in terms of the earth mover's distance and average identity similarity metrics.
3) IDENTITY SIMILARITY, CONDITIONAL EXPERIMENT First, we assess the quality of the proposed Age & Gender Manipulation Module. We demonstrate several examples of manipulation on the FFHQ Aging dataset in Figure  9. The input image is encoded into a latent code by the GAN-inversion encoder E and the corresponding (decoded) image reconstructions are shown in the second column. The model is able to convincingly change the facial appearance to match the specified gender and age group. Next, we design a conditional experiment, where all compared models are given age and gender information, in addition to parent images, as inputs. The results are evaluated in terms of identity similarity. We test the models using both the NoK dataset, which already contains the necessary information, and the FIW dataset, which only includes gender information.
To obtain the age information for the FIW dataset, we use the pre-trained age model A. The synthesized results are analyzed using the identity similarity as computed by the ArcFace model. The identity similarities are again compared in terms of the earth mover's distance and average similarity metrics. Table  3  shows that ChildNet achieves the best results on both datasets. Figure  10  shows example visual results for the NoK dataset, comparing methods with high visual quality. ChildNet achieves the most visually compelling, high-quality child synthesis results that bear high resemblance to the real child images.
4) PERCEPTUAL SIMILARITY In this experiment, we focus on measuring the perceptual quality between the synthesized and the real images. Higher image quality corresponds to less distortion and fewer image artifacts. We measure the perceptual quality using the LPIPS metric  [47]  with the AlexNet backend  [60] , which was shown to align well with human perception of image quality. The results are shown in Table  4. The perceptual differences between the methods using the StyleGAN decoder are relatively small. The StyleDNA model achieves the best results for the FIW dataset, with ChildNet being a very close competitor. ChildNet achieves the best results on the NoK dataset.
B. ChildNet CHARACTERISTICS 1) KINSHIP MODULE ABLATION STUDY We conduct an ablation study of the Kinship Module model by removing certain loss terms and by altering the Kinship  Module architecture. The impact of the individual model setting is measured using the identity similarity as computed by the ArcFace model. The results are reported in terms of the earth mover's distance and average cosine similarity. To accelerate the training convergence, all ablation tests are performed at a reduced model resolution of 256 × 256 and with an increased learning rate of η = 1 • 10 -3. We analyze the impact of the loss terms in the final loss function of the Kinship Module in Eq. (  8 ) on the identity similarity of the synthesized child images as measured by the ArcFace model. For this purpose, we train the Kinship Module on the NoK dataset with different combinations of loss terms.
The results are shown in Table  5. A randomly initialized network already achieves decent results, as the initialized parameter weights tend to average the latent codes of the parents. The lack of the magnitude loss term is not reflected in quantitative results; however, the results show a considerable amount of visual artifacts, as shown in Figure  11. This can be attributed to the unbounded values of residual coefficients ϵ, which can push the latent codes outside the learned StyleGAN latent code distribution. Adding the triplet loss term helps achieve a slightly better identity similarity results without compromising the visual quality. Next, we investigate the impact of the Kinship Module architecture. The following architecture settings are considered: • presence of the Mutation BaseNet, i.
2) AGE & GENDER MANIPULATION MODULE ABLATION STUDY Next, we conduct an ablation study on the Age & Gender Manipulation Module Study by analyzing the impact of the TABLE 5. Ablation study of the Kinship Module loss terms. The ablation study is performed with respect to the earth mover's distance on similarity histograms and the average similarity. We use the ArcFace model to calculate similarities. TABLE  6. Ablation study of the Kinship Module architecture. The ablation study is performed with respect to the earth mover's distance on similarity histograms and the average similarity. We use the ArcFace model to calculate the similarities.
The visual results are shown in Figure  12. We observe that the lack of the magnitude loss causes image artifacts on the target image. Since the residual vector ϵ is not regularized, the predicted latent code may be outside the learned StyleGAN latent code distribution, causing undesired image artifacts when processed with the StyleGAN decoder. Meanwhile, the lack of preservation loss effectively breaks the cycle model architecture. This, in turn, can cause the target image to exhibit undesired changes in the person appearance, such as a change of the skin tone, hair color, or other characteristics. Including both terms ensures the high image fidelity due to the magnitude loss term and the effective retention of the person's facial characteristics due to the preservation loss term.
5) ANALYZING THE INTERPOLATION COEFFICIENTS In this experiment, we analyze the distribution of the interpolation coefficients α as predicted by the Attention BaseNet of the Kinship Module. A distribution of α values centered around 0. 5 would indicate that the module tends to synthesize an average face between parents. The opposite case would indicate that the model emphasizes certain latent features of the parents with greater weight during the synthesis process. We perform the experiment on the test split of the NoK dataset and record the interpolation coefficients for each latent code part (coarse, medium, and fine). The histogram of the coefficients is shown in Figure  16. Interestingly, we find that the interpolation coefficients are distributed near their extreme values of 0 and 1, especially at the coarse level, which affects the most fundamental face appearance.
This could lead to increased storytelling possibilities, and a more diverse representation in film and television. The negative social impacts may manifest in the following ways: • If the method's training data is not diverse and representative, the technology could perpetuate existing social biases. This could result in unfair representation or exclusion of certain ethnicities and cultures. Effors should be made to ensure that the kinship synthesis methods are trained on diverse and inclusive datasets. • The discovery of previously unknown relatives could have psychological effects on individuals. The emotional consequences of such discoveries should be considered. It is crucial for researchers to carefully consider these effects to ensure ethical development and use of this technology.
VIII. CONCLUSION In this paper we present ChildNet, a novel model for kinship face synthesis. It consists of two main modules: the Kinship Module and the Age & Gender Manipulation Module. The Kinship Module utilizes an attention mechanism to predict the child latent code, focusing on parental latent codes and adjusting them using a mutation mechanism. The Age & Gender Manipulation Module serves to control the age and gender of the synthesized child image. ChildNet achieves the state-of-the-art performance with respect to identity similarity while exhibiting high perceptual quality. Furthermore, it demonstrates versatility in terms of synthesizing multiple images per input, controlling the synthesis variability and the dominant parent influence. Finally, due to the limitations of existing kinship datasets for the task of kinship face synthesis, we introduce a novel kinship dataset Next of Kin, which contains high-resolution face images and metadata on kinship relationships along with other facial image characteristics.
Results for Parents With Different Ethnicities: Age and Gender Manipulation:    HeredityGAN: HeredityGAN defined semantic directions in the W + latent code space that correspond to facial characteristics related to the size of facial regions, e. g. , size of eyes, nose, and jaw. To generate the required data for semantic vectors, we synthesize 100, 000 StyleGAN2 images and process them with landmark prediction model  [64]. During the micro fusion step, where the semantic vectors are moved based on dominant and recessive features of the facial attribute, we design a deterministic algorithm instead of a stochastic one; we always choose the projections of the parent that has the dominant features. HeredityGAN proposed a gender disentanglement mechanism where the latent code is moved towards the target gender latent direction to match the child's gender.
We use the InterFace method  [25]  to identify the gender semantic latent vector. For age disentanglement, we use a similar method as for gender disentanglement, but due to the continuous nature of age data, we modify the control mechanism. Specifically, our goal is to determine the magnitude of the latent code step required to synthesize the desired age. To do this, we take an age scalar ρ ∈ N [1,100] , normalize it to the range R [-1,1] , and multiply it by the trainable parameter P age. The resulting scalar is then multiplied by the InterFace-identified age vector, and added to the original latent father vector: w f ← w f + norm(ρ) • P age • v age , (20) where norm defines the age normalization operation and v age is the semantic age vector obtained from InterFace. To determine the P age value, we set it as a trainable parameter and train it on the NoK dataset using Eq.
(  20 ) to synthesize facial images. The training objective was set to the mean squared error loss between the predicted age ρ of a pre-trained age model and the true age ρ. When changing the child gender, the father latent vector is moved as follows: w f ← w f + P gender • v gender , (21) where P gender ∈ {-1, 1} is a value that depends on the desired child gender and v gender is the InterFace  [25]  identified gender semantic vector. The P gender value is obtained by experimenting with several different values and choosing the most appropriate one given the trade-off between expressed target semantics and identity preservation of the target image. StyleDNA: We use the official StyleDNA online code implementation to evaluate the results on FIW dataset. To evaluate performance on the NoK dataset, we re-train the DNANet-based mapping module of StyleDNA.
VOLUME 11, 2023 TABLE 1. 20231 Overview of the existing kinship datasets in comparison to the proposed dataset Next of Kin. The abbreviations describing the Subjects column stand for parent-child (P-C), grandparent-grandchild (GP-GC) and siblings (Sib). For datasets containing images with different resolutions, we report the median resolution in parentheses. The last column indicates the average image quality as evaluated by MANIQA [48] , a no-reference image quality assessment model.
Summary for used software The statistical results of the software used in the round robin test are illustrated in Figure  8. It can be seen that software most commonly used in the round robin test was PFC3D (Itasca Consulting Group, Inc. , 2021) and in-house software. Yade  (Smilauer et al. , 2021)  was used by two groups. Also, LIGGGHTS  (Kloss et al. , 2012) , LAMMPS  (Sandia National Laboratories, 2001; Thompson et al. , 2022) , HiDEM  (Sakaguchi and Nishiura, 2009) , DEPTH  (Chen et al. , 2020; Nishiura et al. , 2021) , and Kratos Multiphysics  (Dadvand et al. , 2010 (Dadvand et al. , , 2013) )  were all used by one group. A brief introduction to the various software chosen for the assigned task follows. The PFC3D is a prominent commercial software manufactured by ITASCA Consulting Group, Inc.