dataesr/scientific-paragraphs-categorization huggingface dataset:
Our model is trained on the multi-speaker database PATS, which was proposed in  Ahuja et al. (2020)  and designed to study gesture generation and style transfer. It includes three main modalities that we are considering in our approach: text semantics represented by BERT embeddings, Mel spectrogram, and 2D upper body poses. Our contributions can be listed as follows: 1. We propose the first approach for zero-shot multimodal style transfer approach for 2D pose synthesis. At inference, an embedding style vector can be directly inferred from multimodal data (text, speech, and poses) of any speaker by simple projection into the embedding style space [similar to the one used in  Jia et al. (2018) ]. The style transfer performed by our model allows the transfer of style from any unseen speakers, without further training or fine-tuning of our trained model.
Thus, it is not limited to the styles of the speakers of a given database. 2. Unlike the work of  Ahuja et al. (2020)  and previous works, the encoding of the style takes into account three modalities: body poses, text semantics, and speech-Mel spectrograms, which are important for gesture generation  (Ginosar et al. , 2019a; Kucherenko et al. , 2019)  and linked to style. We encode and disentangle content and style information from multiple modalities. On the one hand, a content encoder is used to encode a content matrix from text and speech signal; on the other hand, a style encoder is used to encode a style vector from all text, speech, and pose modalities. A fader loss is introduced to effectively disentangle content and style encodings  (Lample et al. , 2017). In the following sections, we first discuss the related works and more specifically the existing behavior style modeling approaches as well as their limitations.
Conditioning on subject labels during training allows the model to learn a variety of realistic speaking styles. VOCA also provides animator controls to alter speaking style, identity-dependent facial shape, and pose (i. e. , head, jaw, and eyeball rotations) during animation. Ginosar et al. (2019a)  proposed an approach for generating gestures given audio speech; however, their approach uses models trained on single speakers. The aforementioned studies have focused on generating non-verbal behaviors (facial expression, head movement, and gestures in particular) aligned with speech  (Neff et al. , 2008; Karras et al. , 2017; Cudeiro et al. , 2019; Ahuja et al. , 2020). They have not considered multimodal data when modeling style as well as when synthesizing gestures. To the best of our knowledge, the only attempts to model and transfer the style from multi-speakers database have been proposed by  Ahuja et al.
(2020 Ahuja et al. ( , 2022)). Ahuja et al. (2020)  presented Mix-StAGE, a speech-driven approach, that trains a model from multiple speakers while learning a unique style embedding for each speaker. They created PATS, a dataset designed to study various styles of gestures for a large number of speakers in diverse settings. In their proposed neural architecture, a content and a style encoder are used to extract content and style information from speech and pose. To disentangle style from content information, they assumed that style is only encoded through the pose modality, and the content is shared across speech and pose modalities. A style embedding matrix whose each vector represents the style associated to a specific speaker from the training set. During training, they further proposed a multimodal GAN strategy to generate poses either from the speech or pose modality.
Model architecture We propose ZS-MSTM (Zero-Shot Multimodal Style Transfer Model), a transformer-based architecture for stylized upper-body gesture synthesis, driven by the content of a source speaker's speech-text semantics represented by BERT embeddings and audio Mel spectrogram-and conditioned on a target speaker's multimodal style embedding. The stylized generated gestures correspond to the style of target speakers that have been seen and unseen during training. As depicted in Figure  1 , the system is composed of three main components: 1. A speaker style encoder network that learns to generate a fixed-dimensional speaker embedding style from a target speaker multimodal data: 2D poses, BERT embeddings, and Mel spectrogram, all extracted from videos in a database. 2. A sequence-to-sequence gesture synthesis network that synthesizes upper-body behavior (including hand gestures and body poses) based on the content of two input modalitiestext embeddings and Mel spectrogram-of a source speaker and conditioned on the target speaker style embedding.
Training regime This section describes the training regime we follow for training ZS-MSTM. We trained our network using the PATS Corpus  (Ahuja et al. , 2020). PATS was created to study various styles of gestures. The dataset contains upper-body 2D pose sequences aligned with corresponding Mel spectrogram and BERT embeddings. It offers 251 hours of data, with a mean of 10. 7 s and a standard deviation of 13. 5 s per interval. PATS gathers data from 25 speakers with different behavior styles from various settings (e. g. , lecturers and TV shows hosts). It contains also several annotations. The spoken text has been transcribed in PATS and aligned with the speech. The 2D body poses have been extracted with OpenPose. Each speaker is represented by their lexical diversity and the spatial extend of their arms.
While in PATS, arms and fingers have been extracted, we do not consider finger data in our study; that is we do not model and predict 2D finger joints. This choice arises as the analysis of finger data is very noisy and not very accurate. We model 11 joints that represent upper body and arm joints. We consider two test conditions: seen speaker and unseen speaker. The seen speaker condition aims to assess the style transfer correctness that our model can achieve when presented with speakers that were seen during training as target style. On the other hand, the unseen speaker condition aims to assess the performance of our model when presented with unseen target speakers to perform zero-shot style transfer. Seen and unseen speakers are specifically selected from PATS to cover a diversity of stylistic behavior with respect to lexical diversity and spatial extent as reported by  Ahuja et al.
Moreover, our approach is zero-shot, thus is capable of transferring the style of unseen speakers. It is not limited to PATS speakers and can produce gesture in the style of any newly coming speaker without further training or fine-tuning, rendering our approaches zero-shot. Behavioral style is modeled based on multimodal speakers' data and is independent from the speaker's identity ("ID"), which allows our model to generalize style to new unseen speakers. We validated our approach by conducting objective and subjective evaluations. The results of these studies showed that ZS-MSTM generates stylized animations that are close to the target style for target speakers that are seen and unseen by our model. The results of our ablation studies (see Table  4 ) suggest that all three modalities (text, audio, and pose) are important for the performance of our ZS-MSTM model in style transfer tasks.
While we have made some strides, there are still some limitations. The main limitation of ZS-MSTM is that it was not evaluated on an ECAs. The main reason is that it was trained on the PATS Corpus, which include 2D poses. The graphical representation of the data as 2D stick figure is not always readable even when being projected on the video of a human speaker. The main reason behind this problem is that the animation is missing information on the body pose in the Z direction (the depth axis). An interesting direction for future work is to extend our model to capture the different gesture shapes and motion. Gesture shapes convey different meanings. For example, a pointing index can indicate a direction. Hand shapes and arm movement can describe an object, an action, etc. Several attempts have looked at modeling metaphoric gestures  (Ravenet et al.
Also, intense peaks corresponding to glutamine and asparagine are observed in all the CTF fibrillar assemblies assigned by identifying the entire spin systems, supporting the notion of Q/N rich region involving the TDP-43 rigid amyloid core based on previous experimental and theoretical studies by Mompeán and coworkers  (Mompeán et al. , 2014; Mompean et al. , 2015; Mompean et al. , 2016b). To obtain information on the secondary structure content of the residues observed in the rigid amyloid cores, we performed a chemical shift analysis. 13 C chemical shifts are sensitive probes to the local environment, and their values can be used to derive the local secondary structure  (Wishart and Sykes, 1994). We compared 13 Cα and 13 Cβ experimental chemical shifts in CTF aggregates to the database reported by Wang and Jardetzki  (Wang and Jardetzky, 2002)  to extrapolate information on the secondary structure.
Attenuated total reflection fouriertransform infrared spectroscopy The ATR-FTIR spectroscopy was performed on a Nicolet 6700 FT-IR spectrometer (Nicolet Instrument, Madison, WI) equipped with a liquid nitrogen-cooled mercury-cadmium-telluride detector (ThermoFisher Scientific, San Jose, CA, United States), with a spectral resolution of 4 cm -1 and a one-level zero filling. The hydrated TDP fibril samples were deposited (10 μl) on a germanium ATR crystal. Excess water was removed with a nitrogen gas flow, and the spectrum was recorded with 200 scans. The spectra were analyzed to determine the secondary structure elements of each protein with an algorithm based on a second-derivative function and a self-deconvolution procedure (GRAMS and OMNIC softwares, ThermoFisher Scientific) to examine the number and wavenumber of individual bands within the spectral range of the amide I band (1700-1,600 cm -1 ).
Acknowledgments We would like to thank  Antoine Dutour  and  Tanguy Lebrun  for their technical help. We thank the  Nouvelle Aquitaine Regional Council, University of Bordeaux , and the  Contrat Plan Etat-Region (CPER)   CampusB Bordeaux  for the acquisition of the NMR equipment. This work has benefited from the  Biophysical and Structural Chemistry Platform at IECB ,  CNRS UAR 3033 ,  INSERM US001 , and the  CNRS  (IR-RMN FR3050 and Infranalytics FR 2054). ALo and AB thank the  Institut Français Danemark  and the  Ambassade de France au Danemark  (program  BLÅTAND ).
Secondly, an Adaptive Feature Balancing Module (AFBM) is developed, which can automatically balance the conflict between ship semantic information and location information. Finally, the detection capabilities of the ship detection model for multi-scale ships are further improved by introducing the Atrous Spatial Pyramid Pooling (ASPP) module. This innovative module enhances the detection model's ability to detect ships of varying scales by extracting features from multiple scales using atrous convolutions and spatial pyramid pooling. PPA-Net achieved detection accuracies of 95. 19% and 89. 27% on the High-Resolution SAR Images Dataset (HRSID) and the SAR Ship Detection Dataset (SSDD), respectively. The experimental results demonstrate that PPA-Net outperforms other ship detection models.
To achieve better SAR ship detection performance, researchers have gradually applied deep learning-based object detection methods and techniques to this field. Deep learningbased ship detection methods require a large amount of data to train the model. However, in the initial stage of SAR ship detection, researchers often face the challenge of a small dataset size. Lu et al. [40]  improved the detection accuracy of ship detection models applied to a relatively small dataset by combining data augmentation and transfer learning methods, achieving a 1-3% improvement. Rostami et al. [41]  proposed transferring knowledge from the electro-optical domain to the SAR domain by learning a shared invariant cross-domain embedding space, enabling electro-optical domain images to be used to train SAR domain object detection models.
Zhang et al. [42]  proposed a few-shot multi-class ship detection algorithm with an attention feature map and multi-relation detector. Truong et al. [43]  constructed a convolutional neural network model using transfer learning techniques. Zhang et al. [44]  built the first publicly available dataset for SAR ship detection, called the SAR Ship Detection Dataset (SSDD). Wei et al. [45]  constructed the High-Resolution SAR Images Dataset (HRSID) for ship detection, and they applied residual structures and feature pyramid networks to build HR-SDNet. Currently, some researchers are focusing on model lightweighting. For example, Jin et al. [46]  introduced an atrous convolution kernel to reduce the number of parameters while keeping the receptive field unchanged. Ma et al. [47]  suggested a compact detection model, which uses lasso regularization to set the unimportant feature parameters to zero, thereby greatly reducing the parameters of You Only Look Once V4 (YOLOV4).
Secondly, the Adaptive Feature Balancing Module (AFBM) is constructed to address the problem that FPN and other feature fusion methods directly combine adjacent features without considering different contributions of input features to the output feature. In addition, to further enhance the ship detection ability of the model for multi-scale ships, the Atrous Spatial Pyramid Pooling (ASPP) structure is introduced. Finally, we combine these three modules with CSPDarknet53 to build a multi-scale ship detection model for SAR complex backgrounds called Pyramid Pooling Attention Network (PPA-Net). The main contributions of this paper are as follows: (1) By analyzing the limitations of existing attention mechanisms in SAR ship detection, we propose a new attention module called PPAM. This module utilizes a pooling structure to reduce the impact of noise and background information on weight generation.
Materials and Methods As shown in Figure  1 , PPA-Net consists of three parts: the backbone structure, the neck structure, and the head structure. The workflow can be divided into three stages. Firstly, the sub-scenes SAR images are input into the backbone structure composed of CSPDarknet53 and PPAM for feature extraction. CSPDarknet53 includes one CBM (Conv + BN + Mish) block and 5 Resblock_body, which contain a large residual edge and small residual edges. The introduction of residual edges can effectively prevent the loss of information of small targets. In addition, to better suppress non-ship features in SAR images, we insert a PPAM after each Resblock_body. PPAM is a newly designed attention module used to suppress the influence of noise and background information in SAR images. Unlike previous works, when designing PPAM, we consider the influence of noise and background information in SAR images on the generation of attention mechanism weights.
Next, the feature map obtained after feature extraction is optimized by AFBM and ASPP. AFBM is a feature fusion module designed to fully combine the semantic and positional information of ships. ASPP captures multi-scale ship information in the image through dilated convolutions with different dilation rates. Finally, the feature map optimized by ASPP and AFBM is further decoded by the head structure with convolution to generate the sub-scenes SAR images with annotation boxes. ASPP and AFBM is further decoded by the head structure with convolution to generate the sub-scenes SAR images with annotation boxes. Figure  1. Overall structure of our proposed method.
Atrous Spatial Pyramid Pooling (ASPP) Detection capabilities of SAR ship detection models should consider variable ship sizes. To enhance the multi-scale ship detection capability of the ship detection model while reducing the loss of feature information, we introduce the ASPP module, as shown in Figure  6. The module has four parallel branches. The four branches contain three atrous convolutions with different dilation rates (rate = 2, 4, 6) and one regular convolution (kernel size = 1). Compared with pooling, atrous convolution has less information loss while obtaining different receptive field information. We apply this with normal convolutions to further integrate the semantic information of the input features. Finally, to make the output features retain as much receptive field information as possible, we further stack the output features of the four branches.
The introduction of parallel convolutional layers in the ASPP module increases the number of network parameters; therefore, to reduce the number of parameters, we introduce a Depthwise Separable Convolution (DSC) to decode ship location (Figure  7 ). DSC divides the traditional convolution process into regional convolution and inter-channel convolution. The regional convolution extracts the features of each channel of the feature layer, and the inter-channel convolution uses a 1 × 1 convolution kernel to integrate these feature channels. The Batch Normalization prevent the ship detection model from overfit- The introduction of parallel convolutional layers in the ASPP module increases the number of network parameters; therefore, to reduce the number of parameters, we introduce a Depthwise Separable Convolution (DSC) to decode ship location (Figure  7 ).
Dataset Introduction and Experimental Configuration SSDD is the first widely used dataset for performance evaluation of ship detection models in the SAR ship detection field. The dataset is made up of ship images captured by synthetic aperture radar (SAR) using different polarization modes and created by professionals familiar with radar principles, target recognition and tagging tools. The dataset  The introduction of parallel convolutional layers in the ASPP module increases the number of network parameters; therefore, to reduce the number of parameters, we introduce a Depthwise Separable Convolution (DSC) to decode ship location (Figure  7 ). DSC divides the traditional convolution process into regional convolution and inter-channel convolution. The regional convolution extracts the features of each channel of the feature layer, and the inter-channel convolution uses a 1 × 1 convolution kernel to integrate these feature channels.
Dataset Introduction and Experimental Configuration SSDD is the first widely used dataset for performance evaluation of ship detection models in the SAR ship detection field. The dataset is made up of ship images captured by synthetic aperture radar (SAR) using different polarization modes and created by professionals familiar with radar principles, target recognition and tagging tools. The dataset includes 1160 SAR images, covering ships of various sizes ranging from a few to hundreds of pixels, and 2578 ships distributed in various sea conditions. Therefore, in this experiment, SSDD is used as one of the datasets to evaluate the performance of PPA-Net. The images in the SSDD dataset are captured by synthetic aperture radars of different satellites, such as Radarsat-2, TerraSAR-X, and Sentinel-1, and include four polarization modes: HH, HV, VV, and VH.
Some of the images in SSDD are shown in Figure  8. includes 1160 SAR images, covering ships of various sizes ranging from a few to hundreds of pixels, and 2578 ships distributed in various sea conditions. Therefore, in this experiment, SSDD is used as one of the datasets to evaluate the performance of PPA-Net. The images in the SSDD dataset are captured by synthetic aperture radars of different satellites, such as Radarsat-2, TerraSAR-X, and Sentinel-1, and include four polarization modes: HH, HV, VV, and VH. Some of the images in SSDD are shown in Figure  8. In recent years, HRSID has also been frequently used to evaluate the performance of ship detection models in SAR ship detection field. HRSID is constructed using synthetic aperture radar with Sentinel-1 and TerraSAR-X, and includes three polarizations: HH, HV, and VV.
HRSID contains a total of 5604 SAR images and 16,951 ships, and the images in the dataset are cropped to 800 × 800 pixels, which is more convenient for model training. In addition, compared to SSDD, HRSID contains more data, which can lead to be training of deep learning-based ship detection models. Some SAR images from HRSID are shown in Figure  9. In recent years, HRSID has also been frequently used to evaluate the performance of ship detection models in SAR ship detection field. HRSID is constructed using synthetic aperture radar with Sentinel-1 and TerraSAR-X, and includes three polarizations: HH, HV, and VV. HRSID contains a total of 5604 SAR images and 16,951 ships, and the images in the dataset are cropped to 800 × 800 pixels, which is more convenient for model training. In addition, compared to SSDD, HRSID contains more data, which can lead to better training of deep learning-based ship detection models.
Some SAR images from HRSID are shown in Figure  9. images in the SSDD dataset are captured by synthetic aperture radars of different satellite such as Radarsat-2, TerraSAR-X, and Sentinel-1, and include four polarization modes: HH HV, VV, and VH. Some of the images in SSDD are shown in Figure  8. In recent years, HRSID has also been frequently used to evaluate the performance o ship detection models in SAR ship detection field. HRSID is constructed using syntheti aperture radar with Sentinel-1 and TerraSAR-X, and includes three polarizations: HH, HV and VV. HRSID contains a total of 5604 SAR images and 16,951 ships, and the images in the dataset are cropped to 800 × 800 pixels, which is more convenient for model training In addition, compared to SSDD, HRSID contains more data, which can lead to be train ing of deep learning-based ship detection models.
Some SAR images from HRSID ar shown in Figure  9. Ship targets from the dataset are classified into large, medium, and small objects based on the proportion of object detection in MS COCO (Microsoft coco: Common Objects in Context)  [56]. Bounding boxes with an area smaller than 32 × 32 pixels correspond to small objects, those with an area between 32 × 32 pixels and 96 × 96 pixels correspond to medium objects, and those with an area larger than 96 × 96 pixels correspond to large objects. Statistical data of the SSDD and HRSID datasets are shown in Table  1. We ensured the fairness and effectiveness of our experiments in three aspects: hardware configuration, hyperparameter setting of the ship detection model, and dataset configuration. Additionally, the evaluation of all ship detection models was conducted on the aforementioned partitioned datasets, where all models used the same training and testing data.
Ablation Experiment and Module Performance Analysis To evaluate the effectiveness of the three modules, we conducted ablation experiments on PPA-Net by removing each module and using it as a baseline to demonstrate the impact of different combinations of these modules on ship detection. The experimental results are shown in Table  2. PPAM was added separately to the backbone of the baseline to suppress the impact of noise in SAR images on ship feature extraction. As shown in the data in Table  2 , the ship detection model with PPAM added achieved an improvement of 3. 85% in AP, 4. 74% in P, 2. 55% in R, and 0. 03 in F1 compared to the baseline. To illustrate the effectiveness of PPAM more intuitively, a visual comparison of the detection results is shown in Figure  10. The image in Figure  10  contains a large amount of noise, which leads to a smaller difference between ships and the surrounding background, affecting the feature extraction capability of the backbone of the ship detection model.