pinecone/core-2020-05-10-deduplication huggingface dataset:
Linking animal movement and remote sensing - mapping resource suitability from a remote sensing perspective

Optical remote sensing is an important tool in the study of animal behavior providing ecologists with the means to understand species–environment interactions in combination with animal movement data. However, differences in spatial and temporal resolution between movement and remote sensing data limit their direct assimilation. In this context, we built a data-driven framework to map resource suitability that addresses these differences as well as the limitations of satellite imagery. It combines seasonal composites of multiyear surface reflectances and optimized presence and absence samples acquired with animal movement data within a cross-validation modeling scheme. Moreover, it responds to dynamic, site-specific environmental conditions making it applicable to contrasting landscapes. We tested this framework using five populations of White Storks (Ciconia ciconia) to model resource suitability related to foraging achieving accuracies from 0.40 to 0.94 for presences and 0.66 to 0.93 for absences. These results were influenced by the temporal composition of the seasonal reflectances indicated by the lower accuracies associated with higher day differences in relation to the target dates. Additionally, population differences in resource selection influenced our results marked by the negative relationship between the model accuracies and the variability of the surface reflectances associated with the presence samples. Our modeling approach spatially splits presences between training and validation. As a result, when these represent different and unique resources, we face a negative bias during validation. Despite these inaccuracies, our framework offers an important basis to analyze species–environment interactions. As it standardizes site-dependent behavioral and environmental characteristics, it can be used in the comparison of intra- and interspecies environmental requirements and improves the analysis of resource selection along migratory paths. Moreover, due to its sensitivity to differences in resource selection, our approach can contribute toward a better understanding of species requirements
A polarimetric Doppler radar time‐series simulator for biological applications

The high mobility of airborne organisms makes them inherently difficult to study, motivating the use of radars and radar networks as biological surveillance tools. While the utility of radar for ecological studies has been demonstrated, a number of challenges remain in expanding and optimizing their use for surveillance of birds, bats and insects. To explore these topics, a Lagrangian simulation scheme has been developed to synthesize realistic, polarimetric, pulsed Doppler radar baseband signals from modelled flocks of biological point scatterers. This radar simulation algorithm is described, and an application is presented using an agent-based model of the nocturnal emergence of a cave-dwelling colony of Brazilian free-tailed bats (Tadarida brasiliensis). Dualpolarization radar signals for an S-band weather surveillance radar are synthesized and used to develop a new extension of the spectral velocity azimuth display for polarimetric roost-ring signature analysis, demonstrating one capability of this simulation scheme. While these developments will have direct benefits for radar engineers and meteorologists, continuing investment in radar
methods such as these will have cascading effects toward improving ecological models and developing new observational techniques for monitoring aerial wildlife
Economical crowdsourcing for camera trap image classification.

Camera trapping is widely used to monitor mammalian wildlife but creates large image datasets that must be classified. In response, there is a trend towards crowdsourcing image classification. For high‐profile studies of charismatic faunas, many classifications can be obtained per image, enabling consensus assessments of the image contents. For more local‐scale or less charismatic communities, however, demand may outstrip the supply of crowdsourced classifications. Here, we consider MammalWeb, a local‐scale project in North East England, which involves citizen scientists in both the capture and classification of sequences of camera trap images. We show that, for our global pool of image sequences, the probability of correct classification exceeds 99% with about nine concordant crowdsourced classifications per sequence. However, there is high variation among species. For highly recognizable species, species‐specific consensus algorithms could be even more efficient; for difficult to spot or easily confused taxa, expert classifications might be preferable. We show that two types of incorrect classifications – misidentification of species and overlooking the presence of animals – have different impacts on the confidence of consensus classifications, depending on the true species pictured. Our results have implications for data capture and classification in increasingly numerous, local‐scale citizen science projects. The species‐specific nature of our findings suggests that the performance of crowdsourcing projects is likely to be highly sensitive to the local fauna and context. The generality of consensus algorithms will, thus, be an important consideration for ecologists interested in harnessing the power of the crowd to assist with camera trapping studies
Satellite remote sensing to monitor species diversity: potential and pitfalls

Assessing the level of diversity in plant communities from field-based data is difficult for a number of practical reasons: (1) establishing the number of sampling units to be investigated can be difficult; (2) the choice of sample design can impact on results; and (3) defining the population of concern can be challenging. Satellite remote sensing (SRS) is one of the most cost-effective approaches to identify biodiversity hotspots and predict changes in species composition. This is because, in contrast to field-based methods, it allows for complete spatial coverages of the Earth's surface under study over a short period of time. Furthermore, SRS provides repeated measures, thus making it possible to study temporal changes in biodiversity. Here, we provide a concise review of the potential of satellites to help track changes in plant species diversity, and provide, for the first time, an overview of the potential pitfalls associated with the misuse of satellite imagery to predict species diversity. Our work shows that, while the assessment of alpha-diversity is relatively straightforward, calculation of beta-diversity (variation in species composition between adjacent locations) is challenging, making it difficult to reliably estimate gamma-diversity (total diversity at the landscape or regional level). We conclude that an increased collaboration between the remote sensing and biodiversity communities is needed in order to properly address future challenges and development
Mining maximal quasi-bicliques: Novel algorithm and applications in the stock market and protein networks

Several real-world applications require mining of bicliques, as they represent correlated pairs of data clusters. However, the mining quality is adversely affected by missing and noisy data. Moreover, some applications only require strong interactions between data members of the pairs, but bicliques are pairs that display complete interactions. We address these two limitations by proposing maximal quasi-bicliques. Maximal quasi-bicliques tolerate erroneous and missing data, and also relax the interactions between the data members of their pairs. Besides, maximal quasi-bicliques do not suffer from skewed distribution of missing edges that prior quasi-bicliques have. We develop an algorithm MQBminer, which mines the complete set of maximal quasi-bicliques from either bipartite or non-bipartite graphs. We demonstrate the versatility and effectiveness of maximal quasi-bicliques to discover highly correlated pairs of data in two diverse real-world datasets. First, we propose to solve a novel financial stocks analysis problem using maximal quasi-bicliques to co-cluster stocks and financial ratios. Results show that the stocks in our co-clusters usually have significant correlations in their price performance. Second, we use maximal quasi-bicliques on a mining protein network problem and we show that pairs of protein groups mined by maximal quasi-bicliques are more significant than those mined by maximal bicliques
Adaptive concept drift detection

An established method to detect concept drift in data streams is to perform statistical hypothesis testing on the
multivariate data in the stream. The statistical theory offers rank-based statistics for this task. However, these statistics depend
on a ﬁxed set of characteristics of the underlying distribution. Thus, they work well whenever the change in the underlying
distribution affects the properties measured by the statistic, but they perform not very well, if the drift inﬂuences the characteristics
caught by the test statistic only to a small degree. To address this problem, we show how uniform convergence bounds in learning
theory can be adjusted for adaptive concept drift detection. In particular, we present three novel drift detection tests, whose test
statistics are dynamically adapted to match the actual data at hand. The ﬁrst one is based on a rank statistic on density estimates
for a binary representation of the data, the second compares average margins of a linear classiﬁer induced by the 1-norm support
vector machine (SVM), and the last one is based on the average zero-one, sigmoid or stepwise linear error rate of an SVM
classiﬁer. We compare these new approaches with the maximum mean discrepancy method, the StreamKrimp system, and the
multivariate Wald – Wolfowitz test. The results indicate that the new methods are able to detect concept drift reliably and that they
perform favorably in a precision-recall analysis.status: publishe
Efficient Mining of Distance-Based Subspace Clusters

Traditional similarity measurements often become meaningless when dimensions of datasets increase. Subspace clustering has been proposed to find clusters embedded in subspaces of high-dimensional datasets. Many existing algorithms use a grid-based approach to partition the data space into nonoverlapping rectangle cells, and then identify connected dense cells as clusters. The rigid boundaries of the grid-based approach may cause a real cluster to be divided into several small clusters. In this paper, we propose to use a sliding-window approach to partition the dimensions to preserve significant clusters. We call this model nCluster model. The sliding-window approach generates more bins than the grid-based approach, thus it incurs higher mining cost. We develop a deterministic algorithm, called MaxnCluster, to mine nClusters efficiently. MaxnCluster uses several techniques to speed up the mining, and it produces only maximal nClusters to reduce result size. Non-maximal nClusters are pruned without the need of storing the discovered nClusters in the memory, which is key to the efficiency of MaxnCluster. Our experiment results show that (i) the nCluster model can indeed preserve clusters that are shattered by the grid-based approach on synthetic datasets; (ii) the nCluster model produces more significant clusters than the grid-based approach on two real gene expression datasets and (iii) MaxnCluster is efficient in mining maximal nClusters
Improving the performance of the iterative signature algorithm for the identification of relevant patterns

The iterative signature algorithm (ISA) has become very attractive to detect co-regulated genes from microarray data matrices and can be a useful tool for the identification of similar patterns in many other kinds of numerical data matrices. Nevertheless, its algorithmic strategy exhibits some limitations since it is based on statistical behavior of the average and considers averages weighted by scores not necessarily positive. Hence, we propose to take the median instead of the average and to use absolutes scores in ISA's structure. Furthermore, a generalized function is also introduced in the algorithm in order to improve its algorithmic strategy for detecting high value or low value biclusters. The effects of these simple modifications on the performance of the biclustering algorithm are evaluated through an experimental comparative study involving synthetic data sets and real data from the organism Saccharomyces cerevisiae. The experimental results show that the proposed variations of ISA outperform the original version in many situations. Absolute scores in ISA are shown to be essential for the correct interpretation of the biclusters found by the algorithm. The median instead of the average turns the biclustering algorithm more resilient to outliers in the data sets. Copyright © 2011 Wiley Periodicals, Inc
A Classification for Community Discovery Methods in Complex Networks

In the last few years many real-world networks have been found to show a
so-called community structure organization. Much effort has been devoted in the
literature to develop methods and algorithms that can efficiently highlight
this hidden structure of the network, traditionally by partitioning the graph.
Since network representation can be very complex and can contain different
variants in the traditional graph model, each algorithm in the literature
focuses on some of these properties and establishes, explicitly or implicitly,
its own definition of community. According to this definition it then extracts
the communities that are able to reflect only some of the features of real
communities. The aim of this survey is to provide a manual for the community
discovery problem. Given a meta definition of what a community in a social
network is, our aim is to organize the main categories of community discovery
based on their own definition of community. Given a desired definition of
community and the features of a problem (size of network, direction of edges,
multidimensionality, and so on) this review paper is designed to provide a set
of approaches that researchers could focus on.Comment: Published in the Statistical Analysis and Data Mining journal,
  Special Issue: Networks. Volume 4, Issue 5, pages 512-546, October 201
Reduced rank ridge regression and its kernel extensions

In multivariate linear regression, it is often assumed that the response matrix is intrinsically of lower rank. This could be because of the correlation structure among the prediction variables or the coefficient matrix being lower rank. To accommodate both, we propose a reduced rank ridge regression for multivariate linear regression. Specifically, we combine the ridge penalty with the reduced rank constraint on the coefficient matrix to come up with a computationally straightforward algorithm. Numerical studies indicate that the proposed method consistently outperforms relevant competitors. A novel extension of the proposed method to the reproducing kernel Hilbert space (RKHS) set‐up is also developed. © 2011 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 4: 612–622, 201
h-plots for displaying nonmetric dissimilarity matrices

Nonmetric pairwise data with violations of symmetry, reflexivity, or triangle inequality appear in fields such as image matching, web mining, or cognitive psychology. When data are inherently nonmetric, we should not enforce metricity as real information could be lost. The multidimensional scaling problem is addressed from a new perspective. I propose a method based on the h-plot, which naturally handles asymmetric proximity data. Pairwise proximities between the objects are defined, though I do not embed these objects, but rather the variables that give the proximity to or from each object. The method is very simple to implement. The representation goodness can be easily assessed. The methodology is illustrated through several small examples and applied to the analysis of digital images of human corneal endothelia. Comparisons with well-known methods show its good behavior, especially with nonmetric pairwise data, which motivate my methodology. Other databases and methods are analyzed in the supporting information
Bayesian Model Averaging in Astrophysics: A Review

We review the use of Bayesian Model Averaging in astrophysics. We first
introduce the statistical basis of Bayesian Model Selection and Model
Averaging. We discuss methods to calculate the model-averaged posteriors,
including Markov Chain Monte Carlo (MCMC), nested sampling, Population Monte
Carlo, and Reversible Jump MCMC. We then review some applications of Bayesian
Model Averaging in astrophysics, including measurements of the dark energy and
primordial power spectrum parameters in cosmology, cluster weak lensing and
Sunyaev-Zel'dovich effect data, estimating distances to Cepheids, and
classifying variable stars.Comment: 16 pages, 5 figures. Invited review article for special issue on
  Astrostatistic
Learning with many experts: model selection and sparsity

Experts classifying data are often imprecise. Recently, several models have
been proposed to train classifiers using the noisy labels generated by these
experts. How to choose between these models? In such situations, the true
labels are unavailable. Thus, one cannot perform model selection using the
standard versions of methods such as empirical risk minimization and cross
validation. In order to allow model selection, we present a surrogate loss and
provide theoretical guarantees that assure its consistency. Next, we discuss
how this loss can be used to tune a penalization which introduces sparsity in
the parameters of a traditional class of models. Sparsity provides more
parsimonious models and can avoid overfitting. Nevertheless, it has seldom been
discussed in the context of noisy labels due to the difficulty in model
selection and, therefore, in choosing tuning parameters. We apply these
techniques to several sets of simulated and real data.Comment: This is the pre-peer reviewed versio
Wedge Sampling for Computing Clustering Coefficients and Triangle Counts
  on Large Graphs

Graphs are used to model interactions in a variety of contexts, and there is
a growing need to quickly assess the structure of such graphs. Some of the most
useful graph metrics are based on triangles, such as those measuring social
cohesion. Algorithms to compute them can be extremely expensive, even for
moderately-sized graphs with only millions of edges. Previous work has
considered node and edge sampling; in contrast, we consider wedge sampling,
which provides faster and more accurate approximations than competing
techniques. Additionally, wedge sampling enables estimation local clustering
coefficients, degree-wise clustering coefficients, uniform triangle sampling,
and directed triangle counts. Our methods come with provable and practical
probabilistic error estimates for all computations. We provide extensive
results that show our methods are both more accurate and faster than
state-of-the-art alternatives.Comment: Full version of SDM 2013 paper "Triadic Measures on Graphs: The Power
  of Wedge Sampling" (arxiv:1202.5230
A Bayesian beta kernel model for binary classification and online learning problems

Statistical Analysis and Data Mining, 7(6), 434-449. Author's accepted manuscript||The article of record may be found at http://dx.doi.org/10.1002/sam.11241Recent advances in data mining have integrated kernel functions with Bayesian
probabilistic analysis of Gaussian distributions. These machine learning approaches
can incorporate prior information with new data to calculate probabilistic
rather than deterministic values for unknown parameters. This paper
extensively analyzes a speci c Bayesian kernel model that uses a kernel function
to calculate a posterior beta distribution that is conjugate to the prior beta distribution.
Numerical testing of the beta kernel model on several benchmark data
sets reveals that this model's accuracy is comparable with those of the support
vector machine, relevance vector machine, naive Bayes, and logistic regression,
and the model runs more quickly than other algorithms. When one class occurs
much more frequently than the other class, the beta kernel model often
outperforms other strategies to handle imbalanced data sets, including undersampling,
over-sampling, and the Synthetic Minority Over-Sampling Technique.
If data arrive sequentially over time, the beta kernel model easily and quickly
updates the probability distribution, and this model is more accurate than an
incremental support vector machine algorithm for online learning.This work was funded in part by the U.S. Army Research, Development and Engineering Command, Army Research Office, Mathematical Science Division, under proposal no. 61414-MA-II
Prediction using hierarchical data: Applications for automated detection of cervical cancer

Although the Papanicolaou smear has been successful in decreasing cervical cancer incidence in the developed world, there exist many challenges for implementation in the developing world. Quantitative cytology, a semi-automated method that quantifies cellular image features, is a promising screening test candidate. The nested structure of its data (measurements of multiple cells within a patient) provides challenges to the usual classification problem. Here we perform a comparative study of three main approaches for problems with this general data structure: (i) extract patient-level features from the cell-level data, (ii) use a statistical model that accounts for the hierarchical data structure, and (iii) classify at the cellular level and use an ad hoc approach to classify at the patient level. We apply these methods to a dataset of 1728 patients, with an average of 2600 cells collected per patient and 133 features measured per cell, predicting whether a patient had a positive biopsy result. The best approach we found was to classify at the cellular level and count the number of cells that had a posterior probability greater than a threshold value, with estimated 61% sensitivity and 89% specificity on independent data. Recent statistical learning developments allowed us to achieve high accuracy
Discovering an active subspace in a single-diode solar cell model

Predictions from science and engineering models depend on the values of the
model's input parameters. As the number of parameters increases, algorithmic
parameter studies like optimization or uncertainty quantification require many
more model evaluations. One way to combat this curse of dimensionality is to
seek an alternative parameterization with fewer variables that produces
comparable predictions. The active subspace is a low-dimensional linear
subspace defined by important directions in the model's input space; input
perturbations along these directions change the model's prediction more, on
average, than perturbations orthogonal to the important directions. We describe
a method for checking if a model admits an exploitable active subspace, and we
apply this method to a single-diode solar cell model with five input
parameters. We find that the maximum power of the solar cell has a dominant
one-dimensional active subspace, which enables us to perform thorough parameter
studies in one dimension instead of five