dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgments We thank  Gaspard Anthoine  for providing us some preliminary comparisons with the libpaillier libraries and  Anthony Martinez  for the libsnark baseline benchmarking. We thank  Jean-Louis Roch  for fruitful exchanges about the parallelization of the Server side and for pointing out [62]. Finally we thank the anonymous referees who greatly helped improve the paper. This material is based on work supported in part by the  Agence Nationale pour la Recherche  under Grants  ANR-21-CE39-0006 Sangria  and  ANR-15-IDEX-0002.
Sites and data We performed analysis at five mature beech forest stands in Germany: Bad Brückenau (BBR), Mitterfels (MTF), Vessertal (VES), Conventwald (COM), and Lüß (LUE) (Table  1 , based on Table  1  from  Lang et al. , 2017). Total soil P stocks (g P m -2 , up to 1 m depth) decrease strongly along the following gradient: BBR (904) > MTF (678) > VES (464) > COM (231) > LUE (164). Soil was sampled up to 1 m depth at each site, with layer depths of 5-10 cm, for the measurement of total C, N, organic and inorganic P, and other physiochemical properties, such as soil texture, pH, and oxalate-extractable Fe and Al. Modified Hedley fractionations  (Tiessen and Moir, 2008)  were conducted on soils from all depths for the measurements of labile Pi (P extractable with anionic resin and Pi extractable with 0.
Acknowledgements. Lin Yu was supported by the  Swedish government -funded " Strategic Research Area Biodiversity and Ecosystems in a Changing Climate" (BECC) program. Silvia Cal-dararu was supported by the  European Research Council (ERC)  under the European Union's  Horizon 2020 "Research and Innovation" program  (QUINCY; grant no. 647204 ). The authors are grateful to  Friederike Lang ,  Jaane Krüger , and other co-workers from the  German Research Foundation  ( DFG) priority research program  " Ecosystem Nutrition " ( SPP1685 ), for measuring and sharing the data, and to  Jan Engel , for technical assistance with developing the code. Financial support. The article processing charges for this openaccess publication were covered by the  Max Planck Society. Review statement. This paper was edited by  Aninda Mazumdar  and reviewed by two anonymous referees.
Code availability. The scientific part of the code is available under the GPL v3 license. The scientific code of QUINCY relies on software infrastructure from the MPI-ESM environment, which is subject to the MPI-M Software License Agreement in its most recent form. The source code is available online (QUINCY model, https://doi. org/10. 17871/quincy-model-2019, Zaehle et al. , 2019) , but its access is restricted to registered users. Readers interested in running the model should request a username and password from the corresponding authors or via the Git repository. Data availability. The data in the literature review are available upon request from the corresponding author. Supplement. The supplement related to this article is available online at: https://doi. org/10. 5194/bg-20-57-2023-supplement.
Context. Radio astronomy is currently thriving with new large ground-based radio telescopes coming online in preparation for the upcoming Square Kilometre Array (SKA). Facilities like LOFAR, MeerKAT/SKA, ASKAP/SKA, and the future SKA-LOW bring tremendous sensitivity in time and frequency, improved angular resolution, and also high-rate data streams that need to be processed. They enable advanced studies of radio transients, volatile by nature, that can be detected or missed in the data. These transients are markers of high-energy accelerations of electrons and manifest in a wide range of temporal scales (e. g. , from milliseconds for pulsars or fast radio bursts to several hours or days for accreting systems). Usually studied with dynamic spectroscopy of time series analysis, there is a motivation to search for such sources in large interferometric datasets.
This requires efficient and robust signal reconstruction algorithms. Aims. To correctly account for the temporal dependency of the data, we improve the classical image deconvolution inverse problem by adding the temporal dependency in the reconstruction problem, and we propose a solution based on deep learning. Methods. We introduce two novel neural network architectures that can do both spatial and temporal modeling of the data and the instrumental response. Then, we simulate representative time-dependent image cubes of point source distributions and realistic telescope pointings of MeerKAT to generate toy models to build the training, validation, and test datasets. Finally, based on the test data, we evaluate the source profile reconstruction performance of the proposed methods and classical image deconvolution algorithm CLEAN applied frame-by-frame.
Introduction Next-generation radio facilities like LOFAR  (van Haarlem et al. 2013) , MeerKAT/SKA  (Booth & Jonas 2012) , ASKAP/SKA  (Dewdney et al. 2009) , and SKA-LOW  (Scaife 2020 ) allow for high spectral, high-rate, improved angular resolutions, and high instantaneous sensitivity. This is a notable improvement for studying transient radio sources via aperture synthesis using radio interferometers. These sources appear and disappear over time and can be randomly distributed in the sky. They are associated with high-energy physical phenomena (e. g. , pulsars, rotating radio transients (RRATs), Solar System magnetized objects, and Lorimer-type bursts;  Lorimer et al. 2007 ) and, more generally, fast radio bursts  (FRBs). Searching for such sources in large datasets produced by these instruments is a new challenge that requires competitive and efficient signal reconstruction algorithms.
We assume each source has an angular scale that is much smaller than the angular resolution brought by the PSF. In addition, we assume that the cube contains mixtures of sources with constant and varying flux densities over time. Their locations on the sky will be random but constant during the observation. Because a radio interferometric dataset provides exact information on the baseline length and orientation for all samples, the morphology and time dependency of h t are known for all t. Finally, we control the noise level σ ϵ to mimic the various quality levels of the observations. Following the discussion in Sect. 2, we are thus in a multiple degradation scenario, as in  Zhang et al. (2018). In this study, for the sake of simplification, we focus on the monochromatic case where the observing frequency is fixed.
Method In the context of the time-series image deconvolution problem (3), a single image deconvolution method, such as CLEAN, can be independently applied in a frame-by-frame approach. However, this method does not capture the temporal structure of the sky. A method capable of dealing with the temporal evolution of both telescope and the sky is required. Given recent successes of neural networks in various restoration tasks, we propose to solve problem (3) based on DL. In our setting, a network realizes the following mapping: X = N(Y, H, σ ϵ ; θ). Its input contains a degraded cube and information about the degradation (H, σ ϵ ) in a non-blind manner. The symbol θ denotes parameters of the network that are learned from training data {Y i , X i , H i , σ ϵ,i }. We propose two implementations of the network N, called 2D-1D Net and Deflation Net, respectively.
GT cubes We suppose the sources to be unresolved points sources in the sky image. The size of the pixel on the sky is fixed to 1. 5 ′′. We assume that within a sky image of size 30 ′ × 30 ′ (i. e. , 1200 × 1200 pixels), at most 30 constant and 2 transient point sources can be placed. This distribution of sources can be considered compatible with shallow imaging like that of MeerKAT. Following this distribution, we generate 39 000 training sky cubes with dimensions 32 × 256 × 256. These data are divided into three equal parts, containing zero, one, or two transient sources per field. Each validation and test set contains 66 cubes following the same distribution of sources, with half of them containing a transient source and the other half containing two. The source peak flux density (i. e. , the value of the pixel of a constant source in the ground truth) is randomly sampled between 1 and 100.
In this case, a trained neural network can deconvolve an image of any size. Regarding our networks, as only the 2D Net module operates deconvolution, we should only retrain this module, and 1D Net can be frozen. From a dataset of ungridded visibilities, it is up to the scientist to decide which support size fits the scientific objective of the data. It should be trained on a support size that corresponds to the final products used (e. g. , image catalog of a survey, transient detection pipeline). Secondly, the total duration of transient events is worth considering. The total observation period and the number of temporal frames (which defined a slice PSF of the PSF cube) determine the upper limits on the entire duration and the shortest timescale of transient events we deal with. As an illustration, for a fixed number of PSF frames over the total observation period, an increase in the latter will increase the integration time of each PSF frame.
In this case, the dirty, GT, and PSF cubes are fourdimensional data structures, adding the frequency dimension. A simple way to realize this extension is (i) to stack several skies at different frequencies in the channel dimension at the input of the 2D Net module and (ii) to stack several PSFs at different frequencies in the channel dimension at the input of the SFT layer. The computation of the average sky and PSF, and the subtraction step in the Deflation Net should be done for each frequency. Our neural network systems considered normalized pixel values; the ground truth images' pixel values were divided by 100 (i. e. , they lie between 0 and 1). In the same spirit, when facing different pixel intensity ranges occurring in other types of datasets, for example the real one, a min-max normalization allows us to bring our methods into conditions similar to those that are tackled in this paper.
Conclusion In this study we dealt with transient source reconstruction in the context of radio interferometric imaging. We formulated this problem as a deconvolution of image time series. We proposed two neural networks to address this task, namely 2D-1D Net and Deflation Net. Thanks to the SFT layer, they can handle multiple PSFs that vary depending on the observed sky positions. They involve the same submodules: 2D Net and 1D Net. The former deconvolves individual frames and the latter enables temporal sky modeling. 2D-1D Net is based on a simple feedforward inference, whereas Deflation Net involves different computational flows. The latter restores the average sky and uses it to isolate transient sources in individual frames. Experiments based on simulated data and metrics measuring temporal profile reconstruction and background denoising demonstrated superior performance of these DL-based methods over CLEAN in the presence of noise.
Deflation Net performs the best, excelling in reconstructing constant sources and background denoising. The ablation study confirms the temporal modeling enabled by 1D Net significantly increases the sky cube reconstruction performance. A116, page 12 of 14 Learning methods adapted to deconvolution, temporal structures of transient sources, and specificities of instrumental response are key elements to efficiently analyzing the images obtained via radio interferometers in the SKA era. For instance, the raw sensitivity of MeerKAT enables deep imaging (i. e. , a typical σ ∼ 10 µJy in 15 min) as well as high cadence imaging capabilities in a large field of view. Being able to deconvolve the data in high-noise regimes efficiently will maximize the chance to discover new transients and overcome the limitations imposed by current deconvolution methods.
Contrastive representation learning has proven to be an effective self-supervised learning method for images and videos. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as positives that should be contrasted with other instances, called negatives, that are considered as noise. However, several instances in a dataset are drawn from the same distribution and share underlying semantic information. A good data representation should contain relations between the instances, or semantic similarity and dissimilarity, that contrastive learning harms by considering all negatives as noise. To circumvent this issue, we propose a novel formulation of contrastive learning using semantic similarity between instances called Similarity Contrastive Estimation (SCE).
Our method computes relations defined as a sharpened similarity distribution between augmented views of a batch. Each view from the batch is paired with a differently augmented query. Our objective function will maintain for each query the relations and contrast its positive with other images or videos. A memory buffer is maintained to produce a meaningful distribution. Experiments on several datasets show that our approach outperforms our contrastive and relational baselines MoCov2  [29]  and ReSSL  [28]  on images. We also demonstrate using relations for video representation learning is better than contrastive learning. Our contributions can be summarized as follows: • We propose a self-supervised soft contrastive learning approach called Similarity Contrastive Estimation (SCE) that contrasts pairs of augmented instances with other instances and maintains relations among instances for either image or video representation learning.
• We demonstrate that SCE outperforms on several benchmarks its baselines MoCov2  [29]  and ReSSL  [28]  on images on the same architecture. • We show that our proposed SCE is competitive with the state of the art on the ImageNet linear evaluation protocol and generalizes to several image downstream tasks. • We show that our proposed SCE reaches state-of-the-art results for video representation learning by pretraining on the Kinetics400 dataset as we beat or match previous top-1 accuracy for finetuning on HMDB51 and UCF101 for ResNet3D-18 and ResNet3D-50. We also demonstrate it generalizes to several video downstream tasks. 2 Related work.
Similarity contrastive estimation Contrastive Learning methods damage relations among instances which Relational Learning correctly build. However, Relational Learning lacks the discriminating features that contrastive methods can learn. If we take the example of a dataset composed of cats and dogs, we want our model to be able to understand that two different cats share the same appearance, but we also want our model to learn to distinguish details specific to each cat. Based on these requirements, we propose our approach called Similarity Contrastive Estimation (SCE). We argue that there exists a true distribution of similarity w * i between a query q i and the instances in a batch of N images x = {x k } k∈{1,. ,N } , with x i a positive view of q i. If we had access to w * i , our training framework would estimate the similarity distribution p i between q i and all instances in x, and minimize the cross-entropy between w * i and p i which is a soft contrastive learning objective: L SC E * = - 1 N N i=1 N k=1 w * ik log ( p ik ).
Ablation study To make the ablation study, we conducted experiments on ImageNet100 that has a close distribution to ImageNet, studied in Sect. 4. 1. 3, with the advantage to require less resources to train. We keep implementation details close to ReSSL  [28]  and MoCov2  [29]  to ensure fair comparison. Dataset ImageNet  [88]  is a large dataset with 1k classes, almost 1. 3M images in the training set and 50K images in the validation set. ImageNet100 is a selection of 100 classes from ImageNet whose classes have been selected randomly. We took the selected classes from  [37]  referenced in Appendix A. Implementation details for pretraining We use the ResNet-50  [89]  encoder and pretrain for 200 epochs. We apply by default strong and weak data augmentations defined in Table  1. We do not use a predictor, and we do not symmetry the loss by default.
strong-γ combines strong-α and strong-β. We performed a study in Table  4  on which data augmentations are needed to build a proper target distribution for the non-symmetric and symmetric settings. We report the Top-1 accuracy on Imagenet100 when varying the data augmentations applied on the online and target branches of our pipeline. For the non-symmetric setting, SCE requires the target distribution to be built from a weak augmentation distribution that maintains consistency across instances. Once the loss is symmetrized, asymmetry with strong data augmentations has better performance. Indeed, using strongα and strong-β augmentations is better than using weak and strong augmentations, and same strong augmentations have lower performance. We argue symmetrized SCE requires asymmetric data augmentations to produce different relations for each view to make the model learn more information.
The effect of using stronger augmentations is balanced by averaging the results on both views. Symmetrizing the loss boosts the performance as for  [2, 27]. Sharpening the similarity distributions The temperature parameters sharpen the distributions of similarity   5. We found the best values τ m = 0. 07 and τ = 0. 1 proving SCE needs a sharper target distribution. In Appendix E, this parameter search is done for other datasets used in comparison with our baselines. Unlike ReSSL  [28] , SCE does not collapse when τ m → τ thanks to the contrastive aspect. Hence, it is less sensitive to the temperature choice.
Comparison with our baselines We compared on 6 datasets how SCE performs against its baselines. We keep similar implementation details to ReSSL  [28]  and MoCov2  [29]  for fair comparison. Small datasets Cifar10 and Cifar100  [90]   Implementation details Architecture implementation details can be found in Appendix D. 1. For MoCov2, we use τ = 0. 2 and for ReSSL their best τ and τ m reported  [28]. For SCE, we use the best temperature parameters from Sect. 4. 1. 1 for ImageNet and ImageNet100 and from Appendix E for the other datasets. The same architecture for all methods is used except for MoCov2 on ImageNet that kept the ImageNet100 projector to improve results. Results are reported in Table  6. Our baselines reproduction is validated as results are better than those reported by 123 Table  6  Comparison of SCE with its baselines MoCov2  [29]  and ReSSL  [28]   the authors.
Transfer learning We study the generalization of our proposed SCE on several tasks: linear transfer learning (Table  9 ), low-shot (Table  10 ), and object detection and instance segmentation (Table  11 ). We use our multi-crop checkpoint pretrained for 200 epochs on ImageNet. Low-shot evaluation Low-shot transferability of our backbone is evaluated on Pascal VOC2007. We followed the protocol proposed by Zheng et al. [28]. We select 16, 32, 64 or all images per class to train the classifier. Our results are compared with other state-of-the-art methods pretrained for 200 epochs in Table  10. SCE is Top-1 for 32, 64 and all images per class and Top-2 for 16 images per class, proving the generalization of our approach to few-shot learning. Linear classifier for many-shot recognition datasets We follow the same protocol as Grill et al.
[2]  and Ericsson et al. [96]  to study many-shot recognition in transfer learning on the datasets FGVC Aircraft  [97] , Caltech-101  [98]   We report the performance of SCE in comparison with state-of-the-art methods in Table  9. SCE outperforms on 7 datasets all approaches. In average, SCE is above all state-ofthe-art methods as well as the supervised baseline, meaning SCE is able to generalize to a wide range of datasets. Object detection and instance segmentation We performed object detection and instance segmentation on the COCO dataset  [94]. We used the pretrained network to initialize a Mask R-CNN  [95]  up to the C4 layer. We follow the protocol of  [39]  and report the Average Precision for detection AP Box and instance segmentation AP Mask. We report our results in Table  11  and observe that SCE is the second best method after Truncated-Triplet  [39]  on both metrics, by being slightly below their reported results and above the supervised setting.
Pretraining dataset To make the ablation study, we perform pretraining experiments on Mini-Kinetics200 [106], later called Kinetics200 for simplicity. It is a subset of Kinet-ics400 [107] meaning they have a close distribution with less resources required on Kinetics200 to train. Kinetics400 is composed of 216k videos for training and 18k for validation for 400 action classes. However, it has been created from Youtube and some videos have been deleted. We use the dataset hosted 1 by the CVD foundation.
Evaluation datasets To study the quality of our pretrained representation, we perform linear evaluation classification on the Kinetics200 dataset. Also, we finetune on the first split of the UCF101 [108] and HMDB51 [109] datasets. UCF101 is an action classification dataset that contains 13k3 different videos for 101 classes and has 3 different training and validation splits. HMDB51 is also an action classification dataset that contains 6k7 different videos from 51 classes with 3 different splits. Pretraining implementation details We used the ResNet3D-18 network [110] following the Slow path of Feichtenhofer et al. [111]. We kept hyperparameters close to the ones used for ImageNet in Sect. 4. 1. 3. More details can be found in Appendix D. 3. We pretrain for 200 epochs with a batch size of 512.
Leveraging contrastive and relational learning As for the image study, we varied λ from Eq. (  8 ) in the set {0, 0. 125,. , 0. 875, 1} to observe the effect of leveraging the relational and contrastive aspects and report results in Table  13. Using relations during pretraining improves the results rather than only optimizing a contrastive learning objective. The performance on Kinetics200, UCF101 and HMDB51 consistently increases by decreasing λ from 1 to 0. 25. The best λ obtained is 0. 125. Moreover, λ = 0 performs better 1 Link to the Kinetics400 dataset hosted by the CVD foundation: https:// github. com/cvdfoundation/kinetics-dataset. than λ = 1. These results suggest that for video pretraining with standard image contrastive learning augmentations, relational learning performs better than contrastive learning and leveraging both further improve the quality of the representation.
0 as it provides best or second best results for all our benchmarks. It improves our baseline by +2. 1p. p. on Kinetics200 and UCF101, and +5. 0p. p. on HMDB51. It outperforms our supervised baseline by +0. 9p. p. on UCF101 and +1. 9p. p. on HMDB51. 123 Action recognition We compare SCE on the linear evaluation protocol on Kinetics400 and finetuning on UCF101 and HMDB51. We kept the same implementation details as in Sect. 4. 2. 1. We compare our results with the state of the art in Table  18  on various architectures. To propose a fair comparison, we indicate for each approach the pretraining dataset, the number of frames and resolution used during pre-training as well as during evaluation. For the unknown parameters, we leave the cell empty. We compared with some approaches that used the other visual modalities Optical Flow and RGB difference and the different convolutional backbones S3D [116] and R(2+1)D-18  [117].