dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments Part of the LC-MS/MS measurements were made in the Molecular Education, Technology, and  Research Innovation Center (METRIC) at North Carolina State University , the ProGénoMIX platform at  CEA-Marcoule  supported by the  IBISA network. Parts of the bioinformatics analysis was carried out using the high-performance computing facilities of the  University of Luxembourg  (https://hpc. uni. lu). Funding work was supported by the  Research Foundation -Flanders (FWO)  [grant no. 1S90918N  to TVDB;  12I5217N  to BM;  G042518N  to LM]; by a  FEBS Summer Fellowship  [to TVDB]; by the  European Union 's  Horizon 2020 Program  ( H2020-INFRAIA-2018-1 ) [823839 to LM]; by the  FEMS [RTG  to SSS]; by the  Norwegian Centennial Chair program  [to TJG, PDJ and MA]; the  Novo Nordisk Foundation  grant  NNF20OC0061313  to MA; the  USDA National Institute of Food and Agriculture Hatch  project  [1014212  to MK]; the  U.
S. National Science Foundation  [OIA 1934844 and IOS 2003107 to MK]; the  Foundation for Food and Agriculture Research  [Grant ID:  593607  to MK]; the  Agence Nationale de la Recherche  [ ANR-17-CE18-0023-01  to GM, OP, JA];  Deutsche Forschungsgemeinschaft (DFG)  [ RE3474/5-1  and  RE3474/2-2  to SF, TM, BYR]. Research by T. J. G. ,P. D. J, E. L were funded by  National Cancer Institute-Informatics Technology for Cancer Research (NCI-ITCR)  grant  1U24CA199347  and  National Science Foundation (U. S. )  grant  1458524  to T. J. G. The European Galaxy server that was used for some calculations is in part funded by  Collaborative Research Centre 992 Medical Epigenetics (DFG  grant  SFB 992/1 2012 ) and the  German Federal Ministry of Education and Research (BMBF  grants  031 A538A/A538C RBC, 031L0101B/031L0101C de.
NBI-epi ,  031L0106  de. STAIR,  031L0103 MetaProtServ (de. NBI)). This work was supported by the  Luxembourg National Research Fund (FNR)  under grants  PRIDE/11823097  and  CORE-INTER/13684739  to BJK, PM and PW, and the  European Research Council  ( ERC-CoG 863664 ) to PW.
Data availability The metaproteomic data sets generated and analyzed in the current study are available via the PRIDE partner repository with the data set identifier PXD023217 (Username: reviewer_pxd023217@ebi. ac. uk Password: DulLrlme). Assemblies and raw metagenomic and metatranscriptomic reads are available through the European Nucleotide Archive under the study accession number PRJEB42466.
76Tandem results have been deposited to the ProteomeXchange Consortium (http://proteomecentral. proteomexchange. org)via the PRIDE partner repository75. SIHUMIx consisted of the combined reference proteomes of the strains extracted from UniProt in July 201976except for Blautia producta, for which the whole genus Blautia was taken (SIHUMIx_REF). The IGC 9. 9 database 77 (available at http://meta. genomics. cn/meta/dataTools) was used as the catalog database for the fecal sample (GUT_REF). Additionally, a meta-omic database from the assembled contigs was produced for both samples using the open reading frame generated with Prodigal (SIHUMIx_MO and GUT_MO). 4 Bioinformatics 4. 1 Generation of protein sequence databases Two types of databases were used for each sample; a catalog (reference) database and a database that was generated from metagenomic and metatranscriptomic (when available) data sequenced from a matching sample (meta-omic database).
The catalog database for The SIHUMIx database (SIHUMIx_REF) is composed of reference proteomes, containing 29,557 proteins (13. 2 MB). In comparison, the metagenomic assembly for SIHUMIx (SIHUMIx_MO) produced 2,719 contigs, with an average contig length of 7. 5 Kbp and the longest contigs being 468 Kbp, yielding 19,319 predicted ORFs (6. 1 MB). For the fecal sample, the IGC reference catalog (GUT_REF) contains 9,879,896 protein sequences (2. 6 GB). The co-assembly of DNA and RNA for the fecal sample (GUT_MO) produced 247,518 contigs with an average length of 1. 6 Kbp and the longest contigs being 600 Kbp. The database GUT_MO yielded protein sequences from 441,558 predicted ORFs (114. 4 MB). All databases were concatenated with a cRAP database of contaminants (https://thegpm. org/cRAP; downloaded in July 2019) and the GUT databases were additionally concatenated with the human UniProtKB Reference Proteome (downloaded in September 2019).
This is the pre-acceptance version, to read the final version please go to ISPRS Journal of Photogrammetry and Remote Sensing. As remote sensing (RS) data obtained from different sensors become available largely and openly, multimodal data processing and analysis techniques have been garnering increasing interest in the RS and geoscience community. However, due to the gap between different modalities in terms of imaging sensors, resolutions, and contents, embedding their complementary information into a consistent, compact, accurate, and discriminative representation, to a great extent, remains challenging. To this end, we propose a shared and specific feature learning (S2FL) model. S2FL is capable of decomposing multimodal RS data into modality-shared and modality-specific components, enabling the information blending of multi-modalities more effectively, particularly for heterogeneous data sources.
Moreover, to better assess multimodal baselines and the newly-proposed S2FL model, three multimodal RS benchmark datasets, i. e. , Houston2013 -hyperspectral and multispectral data, Berlin -hyperspectral and synthetic aperture radar (SAR) data, Augsburg -hyperspectral, SAR, and digital surface model (DSM) data, are released and used for land cover classification. Extensive experiments conducted on the three datasets demonstrate the superiority and advancement of our S2FL model in the task of land cover classification in comparison with previously-proposed state-of-the-art baselines. Furthermore, the baseline codes and datasets used in this paper will be made available freely at https://github. com/danfenghong/ISPRS_S2FL.
• Multimodal RS Benchmark Datasets. Despite the increased availability of the multimodal RS data, different contexts, structures, sensors, resolutions, and imaging conditions pose a great challenge in data acquisition and processing for to-be-studied scenes  (Dalla Mura et al. , 2015). Consequently, the lack of multimodal RS benchmark datasets, to a larger extent, limits the development of the corresponding methodologies and the practical application of land cover classification. • Multimodal Feature Learning Models. Feature extraction (FE), as one of the most important steps prior to the high-level data analysis, also plays a key role in multimodal RS. Most of previous FE methods, either hand-crafted or learning-based  (Rasti et al. , 2020) , usually extract or learn multimodal features in a concatenation fashion.
Such a manner could lead to inadequate information fusion and even hurt or destroy the original components of each modality, due to the highly coupled information between multimodal RS data (particularly heterogeneous data). According to the aforementioned challenges, we aim in this paper to build diversified multimodal RS benchmark datasets and devise novel feature learning models for land cover classification. For this purpose, we first collect multiple RS data with different resolutions, different modalities, and different sensors, with well-labeled land cover maps. Supported by these datasets, we propose a novel multimodal feature learning (MFL) model by decomposing different modalities into shared and specific representations, in order to fuse diverse information from multimodal RS data in a compacted and discriminative way.
The proposed model establishes the explicit mapping relations between the to-be-learned features and original multimodal RS data, providing an interpretable MFL model. More specifically, the contributions of this paper can be highlighted as follows. • Three multimodal RS benchmark datasets are prepared and built with the application to land cover classification. They are diversified, including homogeneous HS-MS Houston2013 datasets, heterogeneous HS-SAR Berlin datasets, and heterogeneous HS-SAR-DSM Augsburg datasets. We will make them freely and openly available after a possible publication, contributing to the RS and information fusion communities. Currently, the heterogeneous RS feature learning and fusion techniques are less investigated, particularly three-modality datasets. To our best knowledge, this is the first time to open heterogeneous RS benchmark datasets that simultaneously involve HS, SAR, and DSM data for land cover classification.
Related Work Image-level fusion is a straightforward way to enhance certain information (e. g. , spatial or spectral resolutions) of homogeneous data, such as RS image pansharpening  (Ehlers et al. , 2010) , HS and MS fusion  (Wei et al. , 2015). However, there are more heterogeneous RS data in reality, typically optical and SAR data that can provide richer and more complementary information. The image-level fusion can not meet the demand for the heterogeneous data fusion task to some extent. Feature-level learning and fusion are needed. For decades, extensive efforts have been made by researchers to develop a variety of MFL algorithms for land cover classification of RS data. These existing methods can be roughly categorized into two main groups from the perspectives of different fusion strategies, i.
, 2019; Yao et al. , 2019) , although a strictly mathematical proof needs to be further improved and perfected. Moreover, we experimentally record the relative loss of the objective function in each iteration to draw the convergence curves of the proposed S2FL model on three datasets used in the experiment section, as shown in Fig. 3. Furthermore, it is clear to observe that the overall computational cost of the optimization problem (1) (i. e. , our S2FL model) is mainly dominated by classic matrix algebra, such as matrix multiplication and matrix inversion. In detail, the update of the linear regression matrix P exhibits a total complexity with O(d 3 s + d 2 s KN + d 2 s C + d s d k K 2 N + d s CKN ). For each iteration of Algorithm 2 in solving the subproblem (6), optimizing Θ 0 and H generally yields the costs of O(d The scene consists of HS and MS data, which is a typical homogeneous dataset.
The original HS image is available from IEEE GRSS data fusion contest 2013 1  and has been widely concerned and applied for land cover classification. This image acquires the campus area of the University of Houston, Texas, USA, with 349 × 1905 pixels and 144 channels covering the spectral range from 0. 38µm to 1. 05µm. To make full use of high spectral information of the HS image and high spatial information of the MS image, we generate the HS-MS Houston2013 benchmark datasets by degrading the original HS image in spatial and spectral domains. More specifically, Spectral degradation: The low spectral resolution MS image can be obtained by using the spectral response functions (SRFs) of the Sentinel-2 sensor. The resulting MS image is composed of the same size with the original HS image and 8 spectral bands at a ground sampling distance (GSD) of 2.
5m. Spatial degradation: The low spatial resolution HS image with a 10m GSD is generated by the means of the bilinear interpolation. To meet the pixel-to-pixel correspondences between HS and MS images, the degraded HS image is re-upsampled to the size of the MS image, i. e. , 349×1905, by using the nearest neighbor interpolation. 3 k K 3 + d 2 k d s K 2 + d 2 k K 3 N + d k d s K 2 N + d k K 3 N 2 ) and O(d 3 s + d 2 s KN + d 2 s C + d s d k KN + d s d k CK + d k K 2 CN + d s CKN ), Table  2  lists the types of ground objects and the number of training and test samples used for the classification task in the HS-MS Houston2013 datasets, and correspondingly Fig. 4  visualizes the false-color images and the distribute of training and test sets.
HS-SAR Berlin Data The simulated EnMAP data synthesized based on the HyMap HS data graphically describes the Berlin urban and its rural neighboring area at 30m GSD, which can be freely downloaded from the website 2. In detail, there are 797 × 220 pixels in this scene, where 244 spectral bands are given in the wavelength range of 0. 4µm to 2. 5µm. More details can be found in  (Okujeni et al. , 2016). To get a corresponding SAR data of the same region, we download a Sentinel-1 dual-Pol (VV-VH) single look complex (SLC) product from ESA. The product is collected under the Interferometric Wide (IW) swath mode. With the help of ESA toolbox SNAP 3 , we build up a  pre-processing workflow to prepare the SLC product as an analysis-ready SAR image. The workflow includes apply orbit profile, radiometric calibration, deburst, speckle reduction, terrain correction, and region-of-interest extraction.
The analysis-ready SAR image used in this paper is geocoded in UTM/WGS84 coordinate system, and it is spatially averaged while applying speckle reduction and saved in 2 × 2 PolSAR covariance matrix. Since the azimuth resolution of the Sentinel-1 data is approximately 13m, the processed SAR image has a 13. 89m GSD and consists of 1723 × 476 pixels. Similarly to the first datasets, the nearest neighbor interpolation is performed on the HS image, enabling the same image size with the SAR data. The ground reference data for land cover classification is generated by using the OpenStreetMap data  (Haklay and Weber, 2008) , where the information regarding the training and test sets is shown in Fig. 4  and Table  3.
HS-SAR-DSM Augsburg Data This dataset consists of three different data sources, including a spaceborne HS image, a dual-Pol PolSAR image, and a DSM image, where HS and DSM data are acquired by DAS-EOC, DLR, and the PolSAR data are collected from the Sentinel-1 platform, over the city of Augsburg, Germany. They are collected by the HySpex sensor  (Baumgartner et al. , 2012) , the Sentinel-1 sensor, and the DLR-3K system  (Kurz et al. , 2011) , respectively. To evaluate the performance of multimodal fusion classification effectively, we downsample the spatial resolution of all images to a unified 30m GSD. The scene comprises of 332 × 485 pixels and 180 spectral bands ranging from 0. 4µm to 2. 5µm for the HS image, 1 band for the DSM image, and 4 features from the dual-Pol (VV-VH) SAR image. Note that the SAR data is preprocessed in the same way as the SAR component in HS-SAR Berlin Data.
Moreover, the feature dimension is also a key parameter, which has great effects on the quality of final learned feature representations. For that, the optimal feature dimension (d) can be found out from 5 to 50 at a 5 interval, according to the best classification performance on the training set. It should be noted, however, that the DSM image only holds one feature band in the Augsburg datasets, hence attribute profiles are first extracted to fully utilize the spatial information of the DSM image, extending the feature bands to 3, before performing feature learning and classification.
Results and Analysis on Houston2013 Datasets Table  5  lists the quantitative classification results of compared feature learning methods on HS-MS Houston2013 datasets, including OA, AA, κ, and the accuracy for each class. Overall, the joint use of multiple modalities (e. g. , HS+MS) obviously performs better than single modalities in three main indices, i. e. , OA, AA, and κ. The classification accuracies obtained by JDR-PCA are basically same to those jointly using HS and MS data. For MA-based approaches (e. g. , USMA, SMA), they fail to classify the materials well, due to their sensitivity to various complex noises. This indirectly leads to relatively poor performance, compared to PCA and HS+MS. Owing to the use of label (supervised) information, the classification accuracy of SMA is higher than that of USMA, bringing an increase of about 2% points OA.
From the overall perspectives, the OA The number of nearest neighbors datasets, e. g. , the number of nearest neighbors and σ in Eq. (  2 ), the regularization parameters α and β in Eq. (  1 ), and the feature dimension. value remains stable as long as these parameters are selected in a proper range. The optimal parameter combination is (q, σ, α, β, d) = (45, 10 0 , 10 -2 , 10 0 , 30), which is basically identical to that obtained by cross-validation on the training set. This, to some extent, demonstrates the acceptable practicability of the proposed S2FL model in the real application.
Ablation Analysis of the Proposed S2FL Model To verify the effectiveness of the proposed idea (i. e. , shared and specific feature disentangling) in the S2FL model, we investigate the performance gain with the use of different components, i. e. , only using modality-shared features (obtained via Θ 0 ), only using modality-specific features (obtained via Θ k ). The quantitative results in terms of OA, AA, and κ indices on the Houston2013 datasets are reported in Table  7. More specifically, the S2FL's results only using one modality (e. g. , HS, MS) 4  are reported, which is better than those without feature learning (the first two rows in Table  7 ). We also consider the case of only using shared features or only using specific features in our S2FL model for land cover classification. As can be seen from Table  7 , the performance of S2FL by considering multimodal input is superior to that of only using single modalities, while the modality-specific information is more important than the modality-shared information (OA: 83.
Results and Analysis on Berlin Datasets Unlike the homogeneous HS-MS data, the heterogeneity between HS and SAR data remains challenging in the feature learning and fusion. Therefore, the quantitative comparison conducted on the Berlin datasets (see  meaningful for the performance assessment of MFL models. As can be seen in Table  6 , the heterogeneous datasets are challenging and difficult to perform the land cover classification, yielding a sharp decrease in classification performance compared to results on the Houston2013 datasets. Nevertheless, the whole trend between different algorithms is similar. The classification accuracy using the concatenation of HS and SAR data is still higher than that only using single modalities. PCA-based joint feature learning obtains a basically same result with HS+SAR's.
Only using SAR data yields a poor classification map with extensive noisy points. Notably, our method is capable of fully blending the HS and SAR information by the means of the interpretable shared and specific feature learning mechanism, thereby reducing the noisy pixels and generating more smooth classification maps. Particularly for those ground objects that hold rich texture information, e. g. , Forest, Low Plants, the S2FL model tends to capture their subtle differences against noises by the means of specific information of each modality. Furthermore, the learned common components can depict the structural information, thereby further identifying the materials of Residential and Commercial effectively.
Results and Analysis on Augsburg Datasets We further investigate the generalization ability and effectiveness of the proposed S2FL model in the case of three-modality data, i. e. , HS, SAR, and DSM. Table  9  details the classification results of different compared algorithms. Generally, there is an obvious improvement (around 4%) in OA when jointly using three modalities, compared to that using HS, SAR, and DSM independently. When the number of considered modalities increases to 3, the performance of those MA-based models dramatically declines, especially USMA that is a lack of label guidance. It is worth noted, however, that the CoSpace-based approaches modeled by either 2 -norm or 1 -norm, can be effectively extended to the case of three modalities, yielding significant improvement in classification accuracies.
Ablation Study on the Use of Multiple Modalities To further verify the effectiveness and superiority of the proposed S2FL model in multimodal RS data feature learning and fusion, we will investigate the performance gain by using different combinations of multiple modalities. More specifically, three high-performance MFL algorithms, i. e. , 2 -CoSpace, 1 -CoSpace, S2FL, are used for quantitative comparison on the HS-SAR-DSM Augsburg datasets, as detailed in Table  10. On the whole, there are several important and intuitive conclusions, which can be summarized as follows: • The joint exploitation of multiple modalities can break the performance bottleneck in land cover classification. For example, the HS+SAR+DSM can usually obtain better classification results than the only use of two modalities.
• Characterized by rich spectral information, the HS image tends to identify the materials at a more accurate level compared to SAR and DSM. • The HS+SAR results obtained by 2 -CoSpace are even slightly better than those of HS+SAR+DSM. This indicates that the 2 -CoSpace method fails to better fuse the multimodal information to some extent when the number of modalities increases. • Feature selection guided by sparsity-promoting 1 -norm is an effective strategy for MFL. The resulting 1 -CoSpace observably outperforms 2 -CoSpace in different modality combinations. • By decoupling the multimodal data into shared and specific components, S2FL is capable of better learning feature representations of multimodal data (cf. It is worth noting that currently-developed DL-based approaches have shown great potential in the fusion and representation learning of multimodal RS data, yet these methods inevitably suffer from various possible performance degradation.
Conclusion Land cover classification has long been considered as a main research topic in the RS and geoscience community. As a crucial step, feature extraction has been paid much attention by researchers. However, the feature representation ability extracted from only single RS data resources remains limited. Fortunately, the rapid development of RS imaging techniques makes multimodal RS data available on a large scale. To speed up the development of multimodal RS data processing and analysis, we in this paper aim at opening three multimodal RS benchmark datasets, they are homogeneous HS-MS Houston2013, heterogeneous HS-SAR Berlin, and threemodality HS-SAR-DSM Augsburg datasets. Further, we also propose a novel MFL model, called S2FL, yielding more discriminative and compact feature blending by learning shared-modality and specific-modality representations.
By comparing with previously-proposed advanced MFL methods, the S2FL model obtains the best classification performance on the three datasets, which is obviously superior to other competitors. We will open the three potential benchmark datasets and the MFL toolbox including newly-proposed S2FL model, contributing to the RS and information fusion community. In future work, we would like to further extend these datasets to a larger scale and also develop the corresponding feature learning models, e. g. , based on more powerful deep learning techniques by embedding more interpretable knowledge or priors to guide the network optimization in the multimodal feature learning task.
Acknowledgement The authors would like to the  Hyperspectral Image Analysis group  at the  University of Houston  and the  IEEE GRSS DFC2013  for providing the  University of Houston HS  dataset. This work is jointly supported by the  German Research Foundation (DFG)  under grant  ZH 498/7-2 , by the  European Research Council (ERC)  under the Euro-pean Union's  Horizon 2020 research and innovation programme  (grant agreement No. [ERC-2016-StG-714087], Acronym:  So2Sat ), by the  Helmholtz Association through the Framework of Helmholtz AI  (grant number:  ZT-I-PF-5-01 ) -Local Unit "Munich Unit @Aeronautics, Space and Transport (MASTr) "  and Helmholtz Excellent Professorship "Data Science in Earth Observation -Big Data Fusion for Urban Research "( W2-W3-100 ) and by the  German Federal Ministry of Education and Research (BMBF)  in the framework of the international future AI lab " AI4EO -Artificial Intelligence for Earth Observation: Reasoning, Uncertainties, Ethics and Beyond " (Grant number:  01DD20001 ), by the  National Natural Science Foundation of China (NSFC)  under grant contracts No.