dataesr/scientific-paragraphs-categorization huggingface dataset:
This loss was calculated considering the probability map output for each pixel being a road. BCE = - 1 N N ∑ i=1 (y i )log(p(y i )) + (1 -y i )log(1 -p(y i )). ( 2 ) where y i is the label (1 for roads and 0 for not road) of the pixel i, and p(y i ) the predicted probability of the pixel to have this label. We also needed to set an optimized number of epochs. An epoch is one learning cycle that takes into account all training images. Figure  7  presents the evolution of the network loss on the training set and on the evaluation set over epochs. The number of epochs should be relative to the size of the training set to avoid over-fitting. As the size of the training dataset is small, this number of epochs should be kept small. To determine the most suitable moment to stop the network (i. e.
The research for an adapted loss is an important perspective of this work. Another important point to note is the difference between loss values for training and validation images. The further these curves are the more over-fitting there is. In this case, the difference was around 10 percent whereas in case of smaller datasets (e. g. , when using the object-based tiling approach, or with a lower overlapping rate between tiles) we have very different loss values between training and evaluation. That again shows the interest of choosing an optimised image construction method.
Results In this section we present some results obtained with the U-Net segmentation of mountain road images. Our model reaches a segmentation accuracy on evaluation (and train) dataset around 60 percent (80 percent) according to the Dice measure and 50 percent (70 percent) according to the IOU measure. Moreover, the error measured by the loss function decreases under 0. 10 (0. 03). Figure  8  shows some of the results obtained on the images of the validation set. The figure shows the initial image on the left, the predicted image in the middle (black and white), and the ground truth image on the right. In most of the images, the road shape is recognizable, and is very similar to the generalised image. It is also often smoother than the initial image, even if it is sometimes less smooth than the generalised one.
Is Data Matching Pre-Process Useful? First, we compare the results of segmentation when the initial images contain all the roads of the initial dataset, even the roads that are removed in the selection process, with our standard results without those roads that are not matched with generalised roads. We can see in Figure  10  that removing the unselected roads from the initial image seems to be a better solution but the results in both cases are quite unsatisfactory for this road. We can note that the network did learn to remove the unimportant roads, but did not do it properly (some road parts remain, and one important road on top is partially erased). Road network selection is based on global metrics in the network  [12] , so we do not expect that a CNN could learn how to select the important roads in a very small extract of the network.
Comparison of Tiling Methods In this subsection we try to compare the two proposed approaches for tile generation (fixed-scale and object-based). We observe in Figure  11  that the fixed-scale tiling approach gives better results than the object-based approach. The object-based approach seems to systematically create loops in bends and enlarge the road symbols too much (it is probably due to its varying scale). This difference could be due to the construction and filtering method. It should be noted that training on the non-filtered dataset gave poor results with both tiling methods. We suppose that the "loop" problem is caused by the coalescence in the initial image, which prevents "seeing" the initial shape of the line. This is why a mixed representation of roads in input images was tried: the shape of the line is displayed in green with a width of 1 pixel, overlayed on top of the larger red symbol that coalesces.
Summary In this subsection we present a summary to show what seems to be the best approach and parameters to derive the training dataset. Table  2  contains the evaluation of each test in terms of Dice value, IOU value, and qualitative comparison with a reference situation (REF). The values of the table are the following: if the predicted images are visually better than the reference prediction, the visual rank is noted (+), if the predicted images are worse than the reference prediction, the visual rank is noted (-), and if the predicted images are similar to the reference prediction, the visual rank is noted (=). The number of images is also given for information. The reference prediction for fixed-size tiling approaches is produced by sliding from the top left corner of the zone a 2. 5 km square tile with an overlap of 40 percent over the roads, with a symbol width representing the importance.
Usefulness of Data Enrichment and Filtering The filtering is clearly necessary, as the use of (almost) blank images in the training process makes a stable loss and bad learning. Another possible filtering process is to exclude tiles that represent complex not mountain situations (such as highway interchange or roundabouts). Our experiments show that this filtering is not useful, as results are similar. In deep learning, the size of the dataset is often an important success criterion. When the available dataset is limited, data augmentation allows us to improve results. We discussed several methods to augment the training dataset in Section 3. 3, and here we present an analysis of the impact of such methods on the results of the segmentation model. The following augmentation processes were tested on the training dataset: First, the dataset produced by the object-based method seems to be the most affected by the data number limitation.
We present in Table  3  the effect of augmentation. In the object-based approach, the small sized crop transformation seems to have a positive impact on the results, it augments the information without hiding the context too much. In all cases, the variability of the evaluation value seems to increase. The rotation seems to augment the quantitative evaluation value, so the prediction is closer to the reference. But it seems to introduce a white spot noise in the images when we visually evaluate the results. Then, in the case of the fixed-size tiling approach, the crop augmentations are not a good option because it distorts the images and reduces the context around the road (Table  4 ). The augmentation by rotation are relevant but do not improve the result. It shows that the problem of our dataset is less the total quantity of images than the diversity of situations visible in images (we lack sinuous roads).
We will investigate in this direction in future research. Our work also experienced limitations that are not errors: • the evaluation measure, and associated loss function should be improved; • our dataset has a limited size and it does not contain enough examples of very narrow and sinuous bend series; • we also faced computation time and memory limitations. Our current examples do not even reach the level of quality of the previous research on buildings  [10]. As explained previously in the paper, the scale in the tile generation process is an important problem. It seems that our training images are much more complex than the training images for building generalisation, which only contain a single building or a building part. Our current images contain too many details that are represented by only a few (fuzzy and crowded) pixels.
This complexity in our input images has at least three consequences:  (1)  we need more examples to train the model; (2) we need larger images or images with a better resolution; (3) the network needs to be much deeper than our current setting if we were to use images of such a complexity. If we want to maintain our current decision to including so many road segments in a single image, we should at least try to increase the image dimensions. Furthermore, in order to learn to displace the roads for example, we only need to know that there is other map objects in close proximity, we do not need a much larger context. We were limited in our experiments by the computational issues, and the time it takes to extend our initial dataset by performing a long manual data matching pre-process. Consequently we did not test these improvements yet.
As this displacement is often caused by the overlap with the symbols of a nearby river or railway, adding this information in the image should improve our results. We might also be able to generalise these rivers and railways in the same time. Furthermore, we showed the interest of several propositions in processing the training dataset, but some other propositions can be tested. For instance, the selection process could be integrated, and other scales and datasets could be used. One of our limitations is the evaluation and the loss function used in the model. Our evaluation method mixing quantitative and qualitative, which provides a good idea of generalisation quality; however, an automated measure integrated in the neural network is preferable. Further work should focus on what a good generalised image is in terms of pixel classification.
g. [3, 36] ), the deleterious effects of TE methylation on gene expression of neighbouring genes in fungal species is understudied, and is limited only to recent studies of the basidiomycete white-rot fungus Pleurotus ostreatus  [19, 37] , in which the correlation between TE methylation and suppression of nearby genes was documented. With the aim to uncover the variation in DNA methylation and its interaction with TEs over short evolutionary time, we used bisulphite-sequencing methodology to generate genome-wide DNA methylation maps at single-base resolution from individuals representing five closely related Neurospora species with the divergence time of 2 MY or less. Furthermore, we gathered data on genomic TE content and gene expression of the same set of strains. Using this broad dataset, we show variation in global DNA methylation levels among the genomes.
The Neurospora methylome data We performed whole-genome bisulphite sequencing of 10 haploid strains of Neurospora. This sample set includes two strains each of Neurospora crassa, N. sitophila and three phylogenetically and reproductively isolated lineages of N. tetrasperma, herein referred to as different species (Supplementary Table  S1 ). Phylogenetic analyses suggest that N. sitophila and N. tetrasperma have diverged more recently than each of them from N. crassa  [38, 39]. The two strains of each species are of different mating type (A or a: Supplementary Table  S1 ); and for each strain, data were collected for tissues from two life stages (i. e. , vegetative and pre-sexual growth) and were analysed as merged datasets. Our sequencing effort resulted in, on average, 25. 2 million paired reads per strain after removing adaptor contaminations (Supplementary Table  S2 ).
Conserved pattern of DNA methylation along chromosomes Despite the variation in global levels of DNA methylation, the overall chromosomal distribution between the investigated genomes was conserved. First, we found a similar sequence context pattern of DNA methylation in Neurospora: in all investigated genomes, mC sites were found in all three sequence contexts: CG, CHG and CHH (where H is A, C or T), but considerably higher methylation occurred in the CHH context (Supplementary Figure  S1 , Supplementary Tables  S8-S9 ). Second, by mapping the distribution of mCs along chromosomes, we found that DNA methylation was relatively (a) Mean genome-wide DNA methylation levels of cytosines (Cs) in the genomes for each strain of Neurospora, as estimated by MethPipe. Each box plot shows the distribution of four data points: the mean methylation levels at cytosine sites within the specific genome from two biological replicates of tissue from each of two life stages.
Hence, datasets as the one generated in this study to address the interaction between DNA methylation and TE activity over a short evolutionary timescale are unique, and provide a novel opportunity to address the causes and consequences of DNA methylation as a mechanism for genome defence. We found the genome-wide DNA methylation level to vary among the investigated species. Nevertheless, a number of traits were found to be conserved among the investigated species of Neurospora, and the patterns are consistent with what is known from other fungi. For example, the overall low methylation levels in the studied Neurospora species (ranging from 2. 5% in N. crassa to 1. 3% in N. tetrasperma) are consistent with previous reports on N. crassa  [30, 31]  and other fungi  [3, 19, 46, [50] [51] [52].
RNA sequencing Sequencing libraries for mRNA were constructed using Illumina TruSeq mRNA library preparation kits (Illumina Inc, Cat# RS-122-2101/2102), including poly-A selection, according to the manufacturer's protocol (#15031047). Sequencing was performed on Illumina Hiseq 2500 platform at the SNP&SEQ Technology Unit to generate 125 bp paired-end reads. RNA-seq data was obtained on the same set of samples as bisulphite data, with one more replicate per treatment (three replicates in total).
Genomic resources For each strain used in this study, we either downloaded the genome sequence and assembly from various genome resources  [41, 69, 70] , or generated the sequence data ourselves. Genome assembly statistics can be found in Supplementary Table  S1. Note that the genome assembly for N. crassa mat a is not available. We sequenced the N. sitophila genomes on the PacBio RSII platform (Pacific Biosciences) at Uppsala Genome Centre (SciLifeLab, Uppsala, Sweden) using four SMRT cells per sample and the C4 chemistry and P6 polymerase (Pacific Biosciences). The strains were prepared as in Svedberg et al. [71]. Raw PacBio sequence data were filtered and assembled using the SMRT Analysis package and HGAP 3. 0 assembler (Pacific Biosciences, https://github. com/ PacificBiosciences/). The assembly size of FGSC5940 (N.
sitophila mat A) is 38 Mb (15 contigs; 7 contigs mapping to the seven chromosomes of the N. tetrasperma L6 mat a reference genome) and FGSC5941 (N. sitophila mat a) is 38. 4 Mb (25 contigs, 7 contigs mapping to same genome as above). Mean coverage of 94. 2X and 97. 5X of was achieved, respectively. To sort and reorder the contigs into scaffolds in N. tetrasperma, genomes from L1 (mat A and mat a), L10 (mat A and mat a), and L6 (mat A) were all aligned to the N. tetrasperma L6 mat a genome using MUMmer (Version 3. 9. 4alpha) [72]. We generated annotations for each genome, except for N. tetrasperma L6 and N. crassa where gff and protein fasta files are available from ENSEMBL Fungi (https://fungi. ensembl. org/ index. html [73] ). Annotations were produced using MAKER  [74]  and functions were assigned to predict proteins using an all-to-all BLAST search against a Neurospora database assembled using available protein sequences from UniProt (https://www.
Analysis of bisulphite sequencing reads The adaptor sequences were removed from the paired-end sequences using TrimGalore (Version 0. 4. 1; parameters: trim_galore --paired --trim1). Quality control was carried out using multiqc (Version 0. 8). The reads were then mapped to the corresponding Neurospora genomes (Supplementary Table  S1 ), except for N. crassa mat a where we used the genome of strain FGSC2489 (mat A) as a reference; as well as Lambda DNA genome (GenBank: J02459. 1), using the Bismark aligner (Version 0. 16. 2)  [78]  with the following parameters: -n 1 -un. The reference genomes were first transformed into a bisulphite-converted version (C-to-T and G-to-A conversions) and then indexed using Bowtie2  [79]. Sequenced reads were also transformed into two versions. After alignment, uniquely mapped reads were reduplicated using the deduplicate_bismark tool.
The average methylation level of each genomic feature (genes/TEs/non-annotated regions) for each life stage was normalized to the number of cytosines per feature in both life stages. We plotted the mean methylation, unless the distribution of the data was apparently not normally distributed (e. g. , zero-inflated probability distribution, or a binomial/multinomial distribution) at which point we show the median. Bisulphite sequencing reads from N. crassa mat a was mapped to the N. crassa mat A assembly due to the lack of an N. crassa mat a assembly. Distribution of methylation levels, GC content, gene and TEs across chromosomes were conducted by dividing the genomes into 10 kb windows, using different functions in BEDTools (Version 2. 26. 0)  [85]. Circos plots were generated using the Bioconductor OmicCircos R package  [86].
Analysis of transcriptome data and connection to methylated regions and TEs RNA reads were first filtered for rRNA content, then trimmed using Trimmomatic  [88]  and mapped to their respective strain genome, except for N. crassa where we used the genome of strain FGSC2489 (mat A) as a reference, using STAR  [89]. Mapping efficiency was 60-70% on average. 'Features counts' from Subread was then used to generate counts at the gene level. Levels of expression were then analysed using the package edgeR in R  [90]. After building the DGEList, counts on CDS were normalized across libraries of all strains, using orthologues. Normalization factors were then used on each strain library. Then, FPKM were computed, allowing comparison of the expression of genes within each strain across treatments. FPKM are normalized by length of genes, so expression can also be compared among different genes, for example, between genes nearby TE and genes that are not.
Acknowledgments We thank  Nahid Heidari  and  Markus Hiltunen  for excellent technical help. Johan Reimegård is financially supported by the  Knut and Alice Wallenberg Foundation  as part of the  National Bioinformatics Infrastructure Sweden at SciLifeLab. This work was funded from  Nilsson-Ehle Endowments and Stiftelsen Lars Hiertas Minne  foundations, and a  European Research Council  grant under the program H2020, ERC-2014-CoG, project  648143  (SpoKiGen; H. J. ).
Data Access All raw sequencing reads generated in this study have been deposited at the Sequence Read Archive as BioProjects PRJNA505300 and PRJNA505304. Genome assemblies based on PacBio reads, genome annotation files, and RepeatMasker TE annotation files have been deposited at Figshare (https://doi. org/10. 6084/m9. figshare. c. 4302929. v1).
(A) The visual display is a stack of three layers: first a natural-like background noise is generated, characterized by noise contrast, mean spatial frequency and bandwidth  [1]. Then, a sample digit is selected from the MNIST dataset  [2] , rectified, multiplied by a contrast factor and overlaid at a random position. Last, a circular, gray mask is put on. (B) The visual display is then transformed in a retinal input which is fed to the "Where" pathway. This observation is generated by a bank of filters whose centers are positioned on a log-polar grid and whose size increases proportionally with the eccentricity. The "Where" network outputs a collicular-like accuracy map. It is implemented by a three-layered neural network consisting of the retinal log-polar input, two hidden layers (fully-connected linear layers combined with a ReLU non-linearity) with 1000 units each.
Visual search task In this manuscript, we built upon an existing model  [14]  by precisely defining the mathematical framework under the Active Inference formalism. This model is based on a simplified generative model for a visual search task and a proposed algorithm to implement the task. First, in order to implement those principles into a concrete image processing task, we construct a simple yet ecological virtual experiment: After a fixation period of 200 ms, an observer is presented with a luminous 128 × 128 display showing a single target overlaid on a realistic noisy background (see Figure  1-A ). This target is drawn in our case from the MNIST database of manuscript digits consisting of 60000 grayscale images of size 28 × 28  [2]. This image is displayed for a short period of about 500 ms which allows to perform (at most) one saccade toward the (unique) target.
Knowing the correct label k t (and position x t ) for this trial t, this network is trained using a gradient descent with a categorical Cross Entropy loss. This loss is by definition: L t K = -log a k t (f t (x)) (1) The gradient descent is computed at each trial and the process is iterated over the set T of trials which give pairs of inputs f t (x) and outputs k t for this supervised learning scheme. The accuracy of this classic neural network is known to exceed 98% over the genuine MNIST database  [2] , on par with human performance. Due to the max-pooling layers used between the convolutional layers, it also shows a robust translation invariance. In our experimental conditions, the network is trained over an augmented MNIST digits dataset, having a variable contrast, a variable shift (from 0 to 15 pixels away from the center) and a variable (randomly generated) background.
Such a computational shortcut is allowed by the independence of the categorical performance with position. Moreover, for each input image, we know the true position in extrinsic (x t ) and intrinsic (u t def. = ρ -1 (x t )) and identify k t of the target. As such, one can compute the average accuracy map over the dataset and optimize equivalently L t S = - u∈S [A 0 (u -u t ) • log w(u|s t ) + (1 -A 0 (u -u t )) • log(1 -w(u|s t )] ( 10 ) where A 0 (∆u) stands for the accuracy map with respect to the true position, that is, of the accuracy when the input image to the "What" pathway is system-atically shifted by ρ(∆u). In our setting, this function varied little for different identities and we averaged it over all possible identities. Combining this translational shift and the shift-dependent accuracy map of the "What" classifier, the actual accuracy map at each trial can be thus predicted under an ergodic assumption by shifting the central accuracy map on the true position of the target (that is with ∆x = ρ(u) -x t ).
Results After training, we observed that the "Where" pathway can correctly predict an accuracy map, whose maximal argument can be chosen to drive the eye toward a new viewpoint with a single saccade. There, a central snippet is extracted, that is processed through the "What" pathway, allowing to predict the digit's label. The full scripts for reproducing the figures and explore the results to the full range of parameters is available at https://github. com/laurentperrinet/ WhereIsMyMNIST (under a GPLv3 license). The network is trained on 60 epochs of 60000 samples, with a learning rate equal to 10 -4 and the Adam optimizer  [22]  with standard momentum parameters. An improvement in convergence speed was obtained by using batch normalization. One full training takes about 1 hour on a laptop.
The code is written in Python (version 3. 7. 6) with the pyTorch library  [23]  (version 1. 1. 0). Saccades distributions and classification success statistics resulting from this simple sequence are presented in Figure  2. Figure  2A-C  provides an example of our active visual processing setup. The initial visual field (Fig. 2A ) is processed through the "Where" pathway, providing a predicted accuracy map (compared with the true accuracy map in Fig. 2B )). The maximal argument of the accuracy map allows to actuate a saccade. The resulting visual field is provided in Fig. 2C , and the classification is done on the central part of the visual field only (red square). To generalize results, 1000 saccades are sampled for different sequences of input visual fields containing a target with a fixed eccentricity, but a variable identity, a variable azimuth and a variable background clutter.