dataesr/scientific-paragraphs-categorization huggingface dataset:
The extracted features and embeddings have the size of the LXMERT hidden size, i. e. 768. During CL, we fix the sample size to 1M examples per CL iteration. Starting from the second CL iteration, 20% of the training sample is sampled from the examples seen in the previous itera-tion. Concerning the number of training examples, it is worth mentioning that we sample the training examples with replacement. Therefore, the number of distinct examples seen by the executor is lower than the sample size. To reduce the complexity of our proposed model, we allow weight sharing between compatible modules. For example, the relateObj and relateSub modules have similar structures and functionalities, both have visual and textual layers to project bounding box features and word embeddings in a multi-modal reasoning space (they also have an output layer to classify the answer).
a top accuracy by 1%, see the CL+L+W. a line in Table 3. However, this improvement has a significant cost, as CL+L+W. a requires 12 training iterations (12M examples) unlike CL+W. a which only needs 4 iterations (4M examples). This reinforces the idea that with a more refined difficulty measure the model has more time to adjust to difficult examples, and its accuracy gradually increases to achieve a better top accuracy in a CL setting. But training on 12M examples is expensive since the overall dataset size is 14M examples. We thus decided to explore different options to obtain comparable results at a lower cost. A promising finding was that pretraining the models for a few iterations with a randomly sampled 1M examples each leads to an accuracy increase of over 1. 5%, as shown by the CL+W.
a+P model which was pretrained for only 2 iterations. This "warms up" the model to the modular aspect of our VQA framework, allowing it to be more general and effective before starting the CL. An interesting finding was that the model reached peak accuracy before iterating over the full CL configuration. The accuracy drop resulting after the 4th iteration may be explained by model overfitting on the questions with 4 objects. Indeed, in the GQA dataset these questions have a substantially unbalanced answer distribution. A further finding is that repeating the same CLiteration twice (as in CL+W. a+P+R) improves the top accuracy results by 1. 1%, while only moderately increasing the number of iterations. This can be explained by the fact that doubling the number of training iterations helps the model better understand the structure of training data without augmenting the Impact of CL.
We perform several experiments to assess the impact of the CL on our compositional visual reasoning framework. We do this by training our model without CL (Unbalanced, Balanced, and Random), then comparing the accuracy performance and the experiment cost in terms of computation cost and training data examples. In Table  4  we report the accuracy and cost results of the conducted experiments and compare them to the performance of our best CL model CL+W. a+P+R. The Unbalanced model (trained on the entire unbalanced training set of 14M) achieves the highest accuracy value of 70. 2%. This model also has the highest training cost among the evaluated models. The Balanced model, trained on the balanced dataset for a large number of epochs, achieves lower results than the Unbalanced model. This is partly due to the fact that the balancing reduces not only the number of questions in the dataset, but also the diversity of the programs.
Also, to the use of the unbalanced testdev-all for evaluation. By comparing our best CL model (CL+W. a+P+R) to the models trained without CL (no-CL), we find very significant gains in terms of computational cost, e. g. an 18-fold reduction compared to the top contender, the model trained on the Unbalanced dataset. The price to pay-a drop of only 2% in accuracy-appears reasonable.
To decide what layers from which modules will share parameters, we assess the similarity between the modules by analyzing their functional and architectural properties. The former is derived from the module reasoning sub-task and the latter is derived from the module layer architectures. We cannot exhaustively describe here all the performed analysis, but in the following, we exemplify our strategy by comparing some of the modules and explaining their inherent similarities and differences. The Select module detects a relevant bounding box (given the name of an object) and the FilterAttr detects a relevant bounding box given an attribute. Functionally, they both solve a detection problem but have different textual argument semantics. Architecturally, they both have the same layer structure: a textual layer, a visual layer, and an output layer.
We decide to share the visual layer between these two modules but use different textual layers to respect the semantic differences between the textual arguments. However, FilterAttr can share its textual layer with other modules having an attribute as a textual argument (VerifyAttr, FilterNot). The Same and Different boolean modules assess whether or not two selected objects share the same characteristic (provided by the textual argument). The probability p of two objects being similar is the opposite of them being different. Therefore, they share the same layers including the output layer and we use the relation p(Different) = 1p(Same) to differentiate them. The object relations modules such as RelateSub and RelateObj have similar functionalities and neural structures. They share their visual layers to get a common scene representation and they share the textual layer due to the semantic similarity of their arguments (a relation).
For the unbalanced version, the train-all split has over 14M examples and the testdev-all has 172,174 examples, while the balanced version has over 943,000 examples for train, 132,062 for val and a testdev of 12,578 examples. To have a larger number of examples available for CL, we use the unbalanced GQA dataset. The exper-iments trained on the balanced dataset use the union of train and val as done in.
When describing the different performed experiments, we use the following notations, which correspond to different algorithmic choices: -Unbalanced: We train on all the examples from the unbalanced GQA train split, we use the traditional random batch training strategy and the model sees all the data examples in every epoch. -Balanced: We train on the balanced version of the GQA dataset. At every epoch, the model is trained on all the balanced dataset examples. -Random: Instead of training on all the dataset examples, only 1M random examples from the unbalanced dataset are presented to the model at every iteration.
Advanced fluorescence imaging techniques such as single-molecule localization microscopy (SMLM) fundamentally rely on the photophysical behavior of the employed fluorophores. This behavior is generally complex and impacts data quality in a subtle manner. A simulation software named Single-Molecule Imaging Simulator (SMIS) is introduced that simulates a widefield microscope and incorporates fluorophores with their spectral and photophysical properties. With SMIS, data collection schemes combining 3D, multicolor, single-particletracking or quantitative SMLM can be implemented. The influence of advanced fluorophore characteristics, imaging conditions, and environmental parameters can be evaluated, facilitating the design of real experiments and their proper interpretation.
These packages are very useful to reproduce some of the artifacts (e. g. , linkage or clustering artifacts) commonly observed in SMLM. However, they use only relatively simplistic descriptions of fluorophore photophysics, generally based on 3-or 4-state models (typically an inactive state, an active state, a dark state and a bleached state). Furthermore, they do not incorporate spectral characteristics of the fluorophores. Thus, existing SMLM simulators do not allow to thoroughly investigate how a fluorophore responds to specific imaging or environmental parameters. Recently the "Virtual SMLM" software has been introduced, which partly addresses this problem by incorporating laser-sensitive phototransformation rates  9. Yet, the photophysical complexity of the fluorophores and their spectral response to light is not taken into account so that crosstalk effects or e.
g. , pH-or triplet-statedependent mechanisms  10, 11  cannot be addressed. More generally, biologists seeking to predict data quality using specific fluorophores and imaging conditions, chemists interested in the design of new dyes or studying their photophysical behavior, and physicists aiming at improved data acquisition schemes would benefit from the possibility to simulate experiments in silico and confront the outcome with real data. A tool to do so has been lacking thus far. Here, the Single Molecule Imaging Simulator (SMIS) package is introduced to simulate a wide variety of SMLM experiments with advanced photophysical and spectral properties of the fluorophores. Central to SMIS is the computation of the photophysical state of each fluorophore with sub-frame time-resolution in response to any data acquisition scheme specified by the user, typically including multiple lasers.
5 kW/cm² (dashed lines) or 0. 5 kW/cm² (dotted lines) readout laser power density, respectively. d Correlation coefficients between rendered images and a ground-truth NPC image with 100% labeling efficiency. e Median merged localization precision retrieved upon processing the SMIS-generated image stacks. Error bars were computed by randomly splitting the dataset in n = 3 subsets and calculating standard deviations between the subsets. Colored dots show individual data points. f Ground-truth photoconversion efficiency. The example above shows that SMIS simulations have the potential to test complex SMLM data collection schemes and possibly hint at optimal parameters. However, the conclusions drawn are only valid as long as the input photophysical models of the used fluorophores are correct.
This derives from the fact that most overlapped localizations were rejected before unmixing, while a high level of background decreased the precision of spot intensity measurement during localization, strongly affecting the retrieved intensity ratios between channels. Thus, the broader univariate histogram as compared to ground truth in Fig. 4c  is essentially a consequence of errors in spot intensity evaluation by localization software. However, it is important to note that the overall quality of the super-resolution images did not necessarily correlate with the level of crosstalk, as emphasized in the case of high localization density, clearly resulting in a degraded image (Supplementary Fig. S14 ). Finally, we interrogated the effect of fluorophore photophysics on crosstalk, by simulating a 2-color dataset employing Alexa647 and Alexa700, as previously described in Lampe et al.
5c ) due to both kinetic exchange and confinement (Supplementary Fig. S18 ). Next we attempted to extract the slow exchange rates between the tightly DNA-bound and the searching states using vbSPT  38. Imposing a 2-state model in vbSPT allowed accurate retrieving of unbiased diffusion coefficients and populations (Fig. 5e ). Interestingly, the accuracy of the recovered exchange rates (Fig. 5f ) was slightly better for mEos3. 2 in the presence of 488 nm light and clearly improved for PA-JF549, showing the advantage of longer tracks even though the number of tracks was lower in PA-JF549 datasets as compared to mEos3. 2 datasets. However when the number of diffusing states was not specified in vbSPT, a false 3-state model was found (Supplementary Fig. S19a ), with exchange rates departing from the ground truth values, suggesting that vbSPT was biased by the strong confinement.
S24. Advanced simulations focused on Förster Resonance Energy Transfer (FRET) between pairs of fluorophores can also be performed. The case of a photochromic single-molecule FRET experiment between EYFP and rsTagRFP is demonstrated in Supplementary Fig. S25. Whereas many useful SMLM simulators have been published in recent years, the simulations presented here all required specific SMIS features. A comparison of the main properties of published simulation software's is presented in Supplementary Table  S15. Clearly, all packages have their advantages and drawbacks. For example, the newly introduced Virtual SMLM  9  is meant to be a real-time simulator where users can change parameters on the fly. Various experimental PSFs can also be used, which are more realistic than the ideal Gaussian or astigmatic PSFs used in SMIS.
On the other hand, fluorophores with their specific spectroscopic and photophysical signatures cannot be input in Virtual SMLM and SPT simulations cannot be performed. In fact, both SMIS and Virtual SMLM offer paradigmshifting properties and complement each other. Supplementary Table  S16  summarizes all microscope-and fluorophore-related properties that can be simulated with SMIS. In total, simulations dealing with 12 different fluorophores, either fluorescent proteins or organic dyes, have been presented in this work. An essential property of SMIS is the possibility for users to incorporate new fluorophores according to specific experiments to be simulated. In doing so, the SMIS fluorophore database can be enriched. Presently, 28 fluorophores are defined, listed in Supplementary Table  S17.
Spectral data from a huge range of fluorophores is available from Web resources such as www. fpbase. org for fluorescent proteins or https://www. thermofisher. com/order/fluorescence-spectraviewer/ for organic dyes, facilitating the introduction of new fluorophores in the SMIS database. For such simulations, simplified ad-hoc photophysical models of the fluorophores are generally sufficient, which may not be real but reproduce experimental behavior reasonably well. For example, in case study #4, such ad-hoc models have been used for the cyanine dyes CF660C and CF680, the photophysical characteristics of which have not been published. Fourth, to cope with the environmental sensitivity of fluorophores, different models for a single fluorophore can be defined based on a common set of photophysical states.
For example, 4 models of Atto647N have been set up in the SMIS fluorophore database (Supplementary Table  S17 ), where the classical 5-state model  41  (fluorescent state, triplet state, anionic radical, cationic radical, bleached state) is implemented with different interconversion rates, reproducing the experimentally observed behavior in PBS, GLOX, STORM, and ROXS buffers. Although many advanced properties of fluorophores can be defined in SMIS, some others, as of today, cannot. This is for example the case of energy transfer mechanisms that have been shown to occur between closely separated fluorophores  42, 43. Unfortunately, accurate simulations of such mechanisms will remain out of reach until a quantitative theoretical foundation becomes available. On the other hand, it has been shown that labels in strong proximity, for example on antibodies, tend to behave like single emitters in dSTORM conditions  44.
Methods Fluorophore's definition and photophysical states. Fluorophores are defined in a SMIS database and can be created or modified by the user in an interactive manner using the "Define new fluorophore" tool. Four types of photophysical states can be specified: fluorescent states, photoactive dark states, non-photoactive dark states, and photobleached states. For each fluorescent state, excitation and fluorescence emission spectra are defined, typically from experimental data available in FPbase (www. fpbase. org) for fluorescent proteins or using e. g. , Spec-traViewer (www. thermofisher. com/order/fluorescence-spectraviewer) for organic fluorophores, in forms of Excel spreadsheets. Absorption spectra from photoactive dark states also need to be specified. If experimental spectra are not available, a tool is available in SMIS to generate tentative model spectra in silico.
tif format) or 3D stacks of images (in Matlab format), in which different parts of the sample (called "subpatterns" in SMIS) are addressed by specific pixel values. This allows to distribute fractions of the fluorophore ensemble to the individual subpatterns. Linkage errors can be specified. When fluorophores are defined to be fluorogenic, a level of fluorogenicity can be defined for each sub pattern, a useful feature to simulate e. g PAINT experiments with fluorogenic chromophores. A set of synthetic virtual samples (e. g. , NPCs, HeLa cell, E. coli bacteria …) can be created by the SMIS "Create virtual samples" tool. Computation of single-molecule diffusion trajectories. Single fluorophores may diffuse in 2D or 3D throughout their associated virtual sample, according to specified diffusion coefficients and/or velocities (in case of directed motion).
The microscope is described by an objective of given numerical aperture NA, which is assumed to give rise to a Gaussian point spread function (PSF) with standard full width at half maximum (FWHM = 1. 22λ/NA where λ is the emission wavelength). The photon collection efficiency takes into account the orientation of the emitting dipoles  46. Transmission efficiencies of inserted emission or dichroic filters (possibly multibands), as well as overall transmission efficiency of the optical setup are also taken into account. Data from commercial filters can be input. The EMCCD detector is described by the effective pixel size, dark current noise, readout noise and gain (counts/detected photon)  3. SMLM data sets with a defined number of frames and frametime and interframetime can then be generated.
SMIS performance. SMIS was developed under Matlab 2022a (https://fr. mathworks. com). The SMIS Graphical User Interface was developed with Matlab App Designer (Supplementary Fig. S26 ). Although SMIS is not meant to behave as a real-time simulation software  9  , it performs reasonably fast (running time of ~1 s/ frame for a typical 3D SMLM experiment involving full photophysics of 3200 mEos4b molecules decorating 100 NPCs on a 2015 high-grade PC (HP Z-book 15 G2 equipped with a i7-4810MQ processor). Complex scenarios may take more time. However, SMIS offers parallel computing to improve performance on multi-CPU computers. Future developments including C++ code will further improve the speed performance of SMIS. As for any simulation software, the produced results merely reflect the input models, and accuracy needs to be traded to some extent for speed.
Those apparent diffusion coefficients were first corrected by subtracting the contribution of localization error (σ²/Δt, where Δt is the frame time and σ is the median localization uncertainty over the considered dataset (~25 nm and ~12 nm for mEos3. 2 and PA-JF549 data, respectively, estimated with Thunderstorm)). Then, unbiased diffusion coefficients D 1 and D 23 were extracted based on the calibration curves of Supplementary Fig. S16e. The fractional population p of weakly DNA-bound molecules relative to all searching molecules was given by: D 23 ¼ p D 1 À Á þ 1 À p À Á D free À Á ð5Þ where D free = 2. 0 μm²/s was assumed to be extracted, as described in Stracy et al. 31  , from calibrated measurements in perturbed cells devoid of nucleoid where NAPs are freely diffusing. Finally p 2 ¼ p p 23 and p 3 ¼ 1 À p À Á p 23.
All figures were generated using ImageJ (https://imagej. nih. gov/ij/) or with homemade routines written in Matlab. Statistics and reproducibility. For all SMIS applications described in this paper, results extracted from processed SMIS-generated image stacks were generally derived from n = 3 simulations carried out for all conditions tested. Whenever relevant, data from single simulations were split in n = 3 subsets to extract standard deviations. Standard deviations were calculated using the std Matlab command. Ground truth data and calibration data shown in the paper were generally derived from n = 1 simulation for all conditions tested. SMIS simulations do not reproduce biological variability and are only affected by variability in fluorophore labeling, stochastic changes in photophysical states or diffusional behavior of the fluorophores, Poissonian noise and detector noise.
Acknowledgements I am indebted to the many people that helped me throughout this work thanks to inspiring discussions, in particular  Leonid Andronov ,  Mark Bates ,  Ulrike Endesfelder ,  Viktorija Glembockyte ,  Juliette Griffié ,  Florian Levet ,  Martin Linden ,  Jonas Ries ,  Daniel Sage ,  Philip Tinnefeld  and  Jochem Vink. I also thank my lab mates for multiple discussions and for testing SMIS and providing suggestions, in particular Jip Wulffele. The SMIS GUI was essentially developed during Covid confinement, but the virus is not acknowledged. This work was supported by the  Agence Nationale de la Recherche  (grants no. ANR-17-CE11-0047-01  and  ANR-20-CE11-0013-01 ).
Code availability The source code, Matlab app, as well as freely usable standalone versions of SMIS vsn2. 1 for Windows can be downloaded at https://mts-commsbio. nature. com/ (Supplementary Software). Versions of SMIS vsn2. 1 for Windows, Linux and Mac OS are available on GitHub at https://github. com/DominiqueBourgeois/SMIS, or can be downloaded via Zenodo (https://doi. org/10. 5281/zenodo. 7503996).
Data availability Simulated datasets generated in this work and corresponding SMIS simulation input files are available from the author on reasonable request. Example SMIS simulation input files for all case studies presented in this article are included in the SMIS example's library provided with the software (Supplementary Software). Source data underlying the graphs presented in the main and supplementary figures of this manuscript can be downloaded at https://mts-commsbio. nature. com/ (Supplementary Data 1).
Survey and study design The authors designed an English-language survey consisting of 21 questions addressing the management of patients with suspected PCNSL and preoperative CST. The questionnaire was developed using the online tool Survey Monkey. Neuro-oncological specialists of 9 European countries (Austria, Czech Republic, France, Germany, Greece, Hungary, Netherlands, Poland, Romania) were asked to distribute the survey amongst their national neuro-oncology centers. The selection of possible participating countries was made considering the collaborative network of the Austrian authors and with the aim to achieve a good representation of Europe. Further, it was important to us to rule out duplicate responses and, accordingly, we did not want to address participating hospitals anonymously but through local distributors with a good local neuro-oncological network and good personal contacts.
Survey content The survey consisted of 21 questions divided into 3 sections. The first section aimed to collect anonymous data of the hospital and the neurosurgical department. The second section addressed the center's specific management of patients with suspected PCNSL and preoperative CST. This section was concluded by 3 illustrative cases with preoperative MRI sequences before and after administration of CST (Fig. 1 ). The final section included questions on surgical and diagnostic methods. The complete survey can be found in the appendix.
Statistical analysis Statistical analysis was performed using IBM SPSS Statistics for windows version 27 (V. 27, version 27, SPSS Inc. , IBM, Chicago, IL, USA). Given normal distribution (tested by Kolmogorow Smirnow test) metric data are described using mean and standard deviation. Skewed data are summarized using median and range. Categorical data are presented as absolute frequencies and percentages. To test for different distribution of nominal variables we used a Fisher exact test. A Mann Whitney U test was used to test for differences in median values of skewed metric data. A p-value <0. 05 was considered to indicate statistically significant results. Fig. 1. Three clinical cases. The first patient received 4 mg Dexamethasone 3 times a day for 7 days after the first MRI (A). The second MRI (B) was performed shortly before the planned biopsy 7 days after the first MRI.