dataesr/scientific-paragraphs-categorization huggingface dataset:
In Bobo-Dioulasso, the good knowledge of the area and the qualitative surveys carried out before these analyses could guide our choice. It should be also noted that the digitalization of all the houses of the non-regular subspace is time consuming. In addition, this sampling frame may require a training for the use of GPS units and the reading of maps. In terms of tools and material, we must stress that the development of this sampling method required the acquisition of relatively expensive data (aerial photographs and especially satellite images), the use of expensive software programmes, such as ArcGis, and specific skills for the analyses. Some of these costs could be reduced by using open-source software, such as QGis, and data provided by the OpenStreetMap community   and by using free-of-charge satellite images, for instance Sentinel-2 imagery.
To compute probabilities, we consider only the result set in which each set of associations contains exactly n sources (one per catalogue, see below for partial matches). For people familiar with databases, it can be seen as the result of inner joins, joining successively each catalogue using a Chi-square criteria. The probabilities we then compute are only based on positional coincidences. Although we show how it is possible to add likelihoods based on photometric considerations, the computation of such photometric likelihoods is beyond the scope of this paper. As the result of a χ -match, two distinct sets of associations may have sources in common: a source having a large positional error in one catalogue may for example be associated to several sources with smaller errors in another catalogue.
9973. In one dimension this value leads to k γ = 3, that is the famous 3σ rule. It means that for 10 000 real associations in a dataset, we theoretically miss 27 of them by applying the candidate selection criterion. From now on we call this crosscorrelation a χ γ -match, or simply a χ -match. In the particular case of two catalogues D M (µ) follows a χ distribution with 2 degrees of freedom -that is a Rayleigh distribution -and k γ=0. 9973 = 3. 443935. This latter value is used in the two-catalogues χ -match of  Pineau et al. (2011b).
General formula Given a set of n candidates from n distinct catalogues satisfying the candidate selection criterion, we know A&A proofs: manuscript no. MultiCatStatXMatch_v1. 02. 1file x, the Mahalanobis distance (Eq. 40), or χ value, which is a real value; s, the result of the selection criterion x ≤ k γ , that is a boolean always equals to true for the sets of associations we keep, so p(s) = 1; -{h i }, i ∈ [1, B n ], the set of B n (Eq. 73) hypotheses to be formulated for each set of association. We then note h k hypotheses in which the n sources are associated with k "real" sources. For a given set of n candidates from n distinct catalogues, the probabilities associated with the various hypotheses are given by Bayes' formula p(h i |x, s) = p(s)p(h i |s)p(x|h i , s) B n k=1 p(s)p(h k |s)p(x|h k , s) , ( 112 ) = p(h i |s)p(x|h i , s) B n k=1 p(h k , s)p(x|h k , s).
Advantage & Limits The main advantage of using p(h i |x, s) is that the likelihoods it is based on do not depend on the positional errors: the only input parameter is the Mahalanobis distance x. Although it is true that x is computed from positional errors, once the χ -match has been performed we do not need the errors anymore: the distributions we use relies only on x. Changing positional errors modifies the priors, not the likelihoods. So we can easily add independent likelihoods based on magnitudes or other parameters. There are two main problems. The first problem is precisely that the likelihoods depend only on x. It means that a set of A&A proofs: manuscript no. MultiCatStatXMatch_v1. 02. 1file very close sources with very accurate positions may have the same probability than a set of distant sources with large positional errors, even if intuitively the risk the first set have to contain spurious association should be far lower than in the second case.
Warning about the non independence of positional uncertainties In surveys providing individual uncertainties, positions of unsaturated bright sources are often more precise than positions of faint sources. The reason has to do with the higher photometric signal-to-noise ratio of bright sources compared to faint sources, while the FWHM is similar. An example of computation of positional uncertainties based on photon statistics can be found for example in the documentation of the SExtractor software  (Bertin & Arnouts 1996). As mentioned in the documentation, the photon statistics based error is a lower value estimate. It means that we cannot blindly assume that the positional uncertainties and photometric quantities like apparent magnitudes are independent. Moreover, if the positional errors of sources are related to their magnitudes and if the magnitudes of the sources in different catalogues are also related, then positional uncertainties in the different catalogues are related too.
Acknowledgements. A large part of this work was supported by the  ARCHES  project. ARCHES (n 0 313146) was funded by the 7 th Framework of the  European Union  and coordinated by the  University of Strasbourg. All figures (except Fig. 1) were made using the ctioga2 plotting program developped by  Vincent Fourmond. F. J. Carrera  also acknowledges financial support through grant  AYA2015-64346-C2-1-P (MINECO/FEDER ). Many thanks to  François Ochsenbein  for pointing out an error in the 1σ conversion of FIRST positional errors (Eq. 25 and  26 ): the  FIRST errors do come  from an ellipse fit, not from two individual 1-dimensional fits of the semi-major and semi-minor axis. Hence the factor is √.
Introduction The increasing brightness of beamlines for macromolecular crystallography (MX) has been a continuing trend in recent years. This, coupled with the development of single-photoncounting pixel detectors and so-called 'shutterless' data collection, has translated into faster data collection and, owing to higher flux densities, the collection of X-ray diffraction data from very small crystals of biological macromolecules. However, because of radiation damage effects, the obtainable resolution of a complete dataset is reduced as the crystal volume becomes smaller. A valuable strategy for overcoming this and the limitations imposed by radiation damage consists of collecting small partial datasets  (Garman, 2010; Owen et al. , 2011)  from a series of crystals and merging these to construct a complete dataset.
ISSN 1600-5767 When a few degrees -or more -of oscillation data per crystal are available, diffraction images can be processed by standard crystallographic software such as XDS  (Kabsch, 2010)  or DIALS  (Waterman et al. , 2013) , and the resulting partial datasets merged to produce the final complete dataset. Here, to achieve the best results, hierarchical cluster analysis (HCA) can be applied to select a suitable subset of the partial datasets for merging. This method, aimed at determining the most isomorphous datasets out of a large number, has already been successfully used  (Giordano et al. , 2012; Foadi et al. , 2013). A complementary approach uses global optimization algorithms, such as genetic algorithms  (Zander et al. , 2016) , to indicate the best grouping of partial datasets in order to achieve the best final statistics possible.
Genetic algorithms, however, rely on hundreds of scaling and merging runs, rather than just the few required for HCA, and are thus more time consuming than HCA, often requiring several hours to converge to a result. More recently, a new algorithm has also been published to distinguish between random and systematic errors and account for the case when datasets are highly partial or weak and thus below the limits of application of HCA  (Diederichs, 2017). In HCA one can use either unit-cell variations  (Foadi et al. , 2013)  or the correlation coefficients (cc  (a,b)  ) between common intensities in different datasets a and b  (Giordano et al. , 2012)  as a metric of non-isomorphism. However, for very small partial datasets unit-cell parameters usually cannot be determined with sufficient accuracy and thus, provided enough partial datasets are available, the use of intensity-based correlation coefficients would seem to be more reliable  (Giordano et al.
, 2012). Here, we present the software ccCluster, the main goals of which are to provide HCA based on cc (i,j) and to provide a graphical user interface (GUI) making the interpretation of, and interaction with, the resulting dendrogram more accessible to users. A major improvement from the previous implementation is that by using ccCluster merging of partial datasets can be directly performed, without manual editing of input files for XSCALE  (Kabsch, 2010)  or POINTLESS  (Evans & Murshudov, 2013) , and multiple thresholds can be rapidly tested and compared via the software interface to achieve the best final statistics. The tools developed can also be used in automated pipelines for protein structure solution using many partial datasets. ccCluster provides both an easy-to-use graphical interface for HCA and a large choice of options for command-line operation.
Program and dependencies ccCluster is written in Python 2. 7, using cctbx (Grosse-Kunstleve et al. , 2002) for crystallographic data manipulation and NUMPY for cluster analysis. The ccCluster GUI has been written in PyQt5, using matplotlib  (Hunter, 2007). A flowchart of how HCA is implemented within ccCluster is presented in Fig. 1. In the last step of the procedure ccCluster calls well established software, in particular XSCALE  (Kabsch, 2010)  for the merging of partial datasets, and in each output folder produces a simple script allowing users to run the program POINTLESS  (Evans & Murshudov, 2013)  in order to produce directly an unmerged mtz file. This can then be used by the program AIMLESS  (Evans & Murshudov, 2013)  to produce reflection data files suitable for downstream processes in CCP4  (Winn et al.
Distance matrix calculation and clustering method HCA requires a definition of distance between all possible pairs of datasets. The calculation of these distances is performed by the ccCalc class in ccCluster. This class has two functions: one for loading all partial datasets to be analysed and the other to calculate the distance between them. The distance, chosen using a command-line option, is defined on the basis of either unit-cell variation or an intensity-based correlation coefficient. For the latter a distance defined by dða; bÞ ¼ 1 À cc 2 Flowchart of HCA using ccCluster. Input files can come from either XDS or DIALS processing. Merging is performed automatically with XSCALE, but a POINTLESS-AIMLESS run is also possible. has proven to be suitable for the selection of partial datsets to merge  (Giordano et al.
, 2012). ccCluster uses the same metric, but instead of relying on cc 2 ða;bÞ as calculated by XSCALE  (Kabsch, 2010) , which are calculated after applying corrections to the individual datasets, this is directly obtained using the cctbx method miller_array. correlation. coefficient. Here, the consistency of unit-cell parameters between datasets a and b is verified with the cctbx function assess_symmetry() and cc 2 ða;bÞ is then calculated from the common reflections in each pair of unmerged datasets. When unit-cell parameters for two datasets are not compatible, i. e. when they differ by more than 1%, their distance is assigned a value of 1, corresponding to a null correlation. This procedure helps in the determination of outliers. As noted above, variation in unit-cell parameters can also be used for HCA of partial datasets in ccCluster.
Here, inspired by BLEND  (Foadi et al. , 2013)  which uses the variation of the unit-cell diagonal, we calculate the distance between datasets from the maximal variation of one of the unit-cell lengths A, B or C: dða; bÞ ¼ max A a À A b min A a ; A b ð Þ ; B a À B b min B a ; B b ð Þ ; C a À C b min C a ; C b ð Þ ! : ð2Þ It is, however, important to note that the unit-cell parameters are highly sensitive to detector distance refinement and that not all three parameters are precisely determined when the diffraction wedges have less than 10 rotation. Thus, in ccCluster a distance based on cc 2 ða;bÞ is set as the default option. The clustering deployed in ccCluster uses the average linkage method, which defines the distance between two clusters X and Y as the average of the distances between all pairs of datasets from the two clusters: DðX; YÞ ¼ 1 N X þ N Y X d a; b ð Þ; a 2 X; b 2 Y; ð3Þ N X and N Y being the number of datasets in clusters X and Y.
Threshold estimation As the aim of HCA as implemented in ccCluster is to produce a complete dataset by merging many partial datasets, ccCluster contains an automatic threshold height determination routine, called 'minimal for completeness'. Once a dendrogram is generated, this routine concatenates all the reflection files from a cluster at a fixed threshold level and calculates the overall completeness of the resulting Miller array. It then gives an estimation for the minimal value of the threshold at which the dataset is more than 98% complete. The completeness level can be tuned by the user if desired. From its definition [equation (3)], the clustering threshold is directly correlated with the expected average cc 2 ða;bÞ between the merged datasets in the cluster. For example, a clustering at 0.
4 will translate to an average cc 2 ða;bÞ of $91% between all the datasets within the selected cluster. Clearly, choosing the lowest threshold possible to obtain the desired dataset completeness should give the highest level of cc 2 ða;bÞ and thus the best merging quality. When operating from the GUI, the desired threshold height can be changed directly from the dendrogram representation by clicking on the dendrogram itself. This allows users to rapidly perform multiple merging tests, using different threshold levels, in order to achieve optimal merged dataset quality. A simplified threshold estimation is in any case performed when the program is launched, to give the user some idea of an acceptable clustering strategy. This simpler routine, faster than the minimal threshold for completeness, computes the increase in number of datasets in the largest cluster as a function of the threshold.
GUI description Rapid user interaction is highly desirable when evaluating the effects of choosing different HCA linkage thresholds for partial dataset merging. To this end we have developed a GUI (Fig. 2 ) which can be launched after an initial HCA run. The main panel (Fig. 2a ) of the GUI displays the dendrogram itself as well as mouse-clickable buttons for launching the merging procedure and setting/unsetting the 'anomalous' flag. Another checkbox allows the choice between merging only the largest cluster at a certain threshold (default) or all clusters below this threshold. The results panel (Fig. 2b ) of the GUI gives the user a quick overview of the quality of merged datasets. Along with a picture of the dendrogram and an extraction of the XSCALE. LP statistics, it is possible to plot the values for CC 1/2  (Karplus & Diederichs, 2012) , sigAno (|F + À F À |/) and hI/ (I)i as a function of resolution.
Example of SSX data clustering To illustrate the application of ccCluster to serial crystallography data, partial datasets, each comprising 2 of diffraction data with an oscillation range of 0. 1 , were collected at the ESRF beamline ID29 (De  Sanctis et al. , 2012)  from 200 microcrystals (smaller than 20 mm in the largest dimension) of thaumatin contained in a single sample holder. Of the 200 partial datasets collected, 184 were successfully integrated using XDS and were then used as input for ccCluster. Each dataset contained on average 2483 reflections and had an average overall completeness of 4. 9%.
GUI processing and distance definition comparison Wedges containing only 2 of diffraction data present a rather difficult case for cluster analysis. The unit-cell parameters cannot be determined with sufficient precision and the calculation of intensity-based correlation coefficients is adversely affected by the low number of common reflections between each wedge. To test the performance of both approaches, two HCA runs were carried out: one using intensity-based correlation coefficients, the other based on variation of unit-cell dimensions. For HCA using cc (a,b) , automatic analysis in ccCluster suggested the merging of 123 datasets clustering at a linkage distance of 0. 25, with subsequent visual analysis of the dendrogram via the ccCluster GUI suggesting the merging of partial datasets from a smaller cluster (98 datasets) with a linkage distance of 0.
21 (Fig. 3 ). The partial datasets in the smaller cluster were thus merged and scaled (Table  1 ). Subsequently structure solution was carried out using molecular replacement in DIMPLE (http:// ccp4. github. io/dimple/) and model refinement (   higher linkage distance of 0. 8 (Table  1 ) and used the resulting dataset for structure solution and refinement (Table  1 ). HCA using variation of unit-cell dimensions presented a clear distinction between partial dataset subgroups (Fig. 3b ). In this case, the automatic threshold (0. 27) suggested by ccCluster led to the merging and scaling of 90 partial datasets (Table  1 ), with the final dataset also used for structure determination and refinement as outlined above. As can be seen from Table  1 , all the final datasets allowed successful structure solution and refinement.
As might be expected, choosing which partial datasets to merge using HCA based on either cc  (a,b)  or variation of unit-cell dimensions produced both better quality datasets and better final refined models than merging partial datasets indiscriminately. However, it is also clear from Table  1  that both dataset and final refined model quality are better when the choice of partial dataset merging is directed by HCA based on cc (a,b) than they are when HCA is based on variation of unit-cell dimensions. For the ensemble of partial datasets described above, running ccCluster with the 'minimal threshold for completeness' option results in a linkage threshold estimation of 0. 2, very close to the 0. 21 chosen from manual inspection of the dendrogram. This threshold choice resulted in the merging of 92 datasets, producing a final dataset with almost identical characteristics to that produced by visual inspection of the dendrogram (Table  1 ).
To evaluate the efficiency of the -m option, ccCluster was used, employing the -t command line option, to merge partial datasets clustering at various linkage threshold levels, ranging from 0. 05 to 1. 0 in steps of 0. 05. The results of this exercise are shown in Fig. 4. As can be seen, $100% completeness of the resulting dataset is achieved only when the linkage distance used is 0. 2 or above. As might be expected, merging partial datasets clustering at linkage distances higher than 0. 2 results in compiled datasets with slightly higher hI/(I)i, probably due to the increased multiplicity of the final datasets. However, even here there is no improvement in hI/(I)i above a linkage   Use of ccCluster using the -m option and 184 2 wedges collected from different thaumatin crystals. Here the minimal threshold for 98% completeness is estimated to be 0.
2. As outlined in the main text, merging of partial datasets clustering at linkage distances higher than 0. 2 results in compiled datasets with slightly higher hI/(I)i, probably because of the increased multiplicity of the final datasets. However, there is no improvement in this metric above a linkage threshold of $0. 5 as the inclusion of non-isomorphous datasets begins to have an adverse effect on data quality. threshold of $0. 5 as the inclusion of non-isomorphous datasets begins to have an adverse effect on data quality.
Application to data from a sulfur-SAD experiment The application of ccCluster described above concerns the use of HCA to compile a complete dataset from small wedges of data collected from many different crystals. While this is the main intended application of ccCluster, the program is also clearly applicable to the HCA of complete datasets collected from different crystals of the same target. An example of such a use of ccCluster is in the compilation of high-multiplicity datasets such as those required in S-SAD experiments  (Olieric et al. , 2016). Fig. 5  shows the HCA [cc (a,b) ], using ccCluster, of nine individual datasets (supporting information, Table  S1 ) collected from crystals of tetragonal lysozyme using X-rays of = 2. 0 A ˚at ESRF beamline ID29. Here, none of the individual datasets could be used for successful S-SAD structure determination using default parameters in hkl2map  (Pape & Schneider, 2004 ) (Fig.
6a ) nor could a dataset compiled by merging all nine datasets (Fig. 6d ). The ccCluster HCA dendrogram shows that the datasets can be split into two groups of 5 and 4 datasets, respectively, one at a linkage threshold of 0. 64 (Fig. 6c ) and another at a threshold of 0. 83 (Fig. 6b ). Complete datasets were thus generated by the merging of the datasets in each of these two clusters (Table  2 ), and these were used in the automated SAD pipeline crank2 (Skubak & Pannu, 2013), with successful structure determination achieved using both datasets. However, they produced slight differences in the completeness of the final model that could be built automatically. As a comparison, we also performed cluster analysis based on unit-cell parameters, for which the dendrogram is shown in Fig. 5(b ). We can observe how one obtains the same two clusters containing the same datasets, thus leading to identical results in the phasing process.
Thus, for this case the fact that the clustering is based on the unit-cell variation or the computer programs  correlation coefficient does not make any significant difference to the results obtained. In this example, the best results for SAD structure solution are obtained with the cluster with the linkage threshold value 0. 64 (Fig. 5a ). It may seem counterintuitive that merging datasets with cc (a,b) as low as 77% (equivalent to a linkage threshold of 0. 64) could improve the anomalous signal required for SAD structure solution. However, the cc (a,b) used in ccCluster is calculated over the whole common resolution range of the datasets collected, and the HCA linkage distances obtained could be dominated by the higher-resolution data shells. Indeed, if we limit our analysis of these S-SAD datasets to a common resolution of 2.
5 A ˚(see supporting information, Fig. S3 ) the linkage HCA distance for the main cluster drops to $0. 32, corresponding to hcc (a,b) i of $94%. This shows that at intermediate resolution the datasets in this cluster are more similar to each other than is suggested by including the whole common resolution range in cc-based HCA. As it is usually lower-resolution data that are used to kick-start SAD structure solution processes, this clearly explains why merging of the five datasets in this cluster makes structure solution much more straightforward and suggests that for SAD structure solution protocols exploiting multi-crystal data collection the use of HCA to guide the compilation of final datasets should perhaps best be carried out at resolutions significantly lower than the maximum resolution obtained.
Conclusions Here we have presented ccCluster, a software aimed at facilitating the application of HCA in MX experiments. We are confident that the user-friendliness of ccCluster, in particular in its GUI mode of operation, will lead to increased and more successful use of HCA in multi-crystal MX. While we have presented two examples as to how ccCluster can be used to rapidly perform HCA, to present results and to compile complete datasets, a detailed analysis of the applicability of HCA in multi-crystal MX is clearly beyond the scope of this article and we refer readers to earlier discussions in this regard  (Giordano et al. , 2012; Foadi et al. , 2013; Zander et al. , 2016 Zander et al. , , 2015)). This software has already been installed at the ESRF MX beamlines and used within the context of the SSX BAG for one year.