Categorical variables are a natural choice for representing discrete structure in the world. However, stochastic neural networks rarely use categorical latent variables due to the inability to backpropagate through samples. In this work, we present an efficient gradient estimator that replaces the non-differentiable sample from a categorical distribution with a differentiable sample from a novel Gumbel-Softmax distribution. This distribution has the essential property that it can be smoothly annealed into a categorical distribution. We show that our Gumbel-Softmax estimator outperforms state-of-the-art gradient estimators on structured output prediction and unsupervised generative modeling tasks with categorical latent variables, and enables large speedups on semi-supervised classification.
For unlabeled data, difficulties arise because the categorical distribution is not reparameterizable. @cite_0 approach this by marginalizing out @math over all classes, so that for unlabeled data, inference is still on @math for each @math . The lower bound on unlabeled data is:
@cite_0
2122262818
We consider statistical inference for the explained variance @math under the high-dimensional linear model @math in the semi-supervised setting, where @math is the regression vector and @math is the design covariance matrix. A calibrated estimator, which efficiently integrates both labelled and unlabelled data, is proposed. It is shown that the estimator achieves the minimax optimal rate of convergence in the general semi-supervised framework. The optimality result characterizes how the unlabelled data affects the minimax optimal rate. Moreover, the limiting distribution for the proposed estimator is established and data-driven confidence intervals for the explained variance are constructed. We further develop a randomized calibration technique for statistical inference in the presence of weak signals and apply the obtained inference results to a range of important statistical problems, including signal detection and global testing, prediction accuracy evaluation, and confidence ball construction. The numerical performance of the proposed methodology is demonstrated in simulation studies and an analysis of estimating heritability for a yeast segregant data set with multiple traits. Like other machine learning paradigms, multi-label learning also suffers from the curse of dimensionality problem. Multi-label dimensionality reduction can alleviate the problem but they generally ask for sufficient labeled samples. Nevertheless, we often may only have scarce labeled samples and abundant unlabeled samples. In this paper, we propose a @math emi-supervised @math ulti-label @math imensionality @math eduction based on @math ependence @math aximization approach (SMDRdm in short). SDMRdm assumes the semantic similarity and feature similarity of multi-label samples are inter-depended. SMDRdm first applies label propagation on a neighborhood graph composed with labeled and unlabeled samples to obtain the soft labels of unlabeled samples, and then measures the semantic similarity between all the training samples (including unlabeled ones) based on these soft labels and available labels of labeled samples. Next, it measures the feature similarity between samples in the subspace projected by the target projective matrix, instead of the original high-dimensional feature space. After that, it maximizes the dependence between these two types of similarities and incorporates the dependence into linear discriminant analysis to optimize the target projective matrix. Experiments on publicly accessible multi-label data sets demonstrate that SMDRdm achieves more prominent results than other related approaches across various evaluation metrics. In addition, the empirical study also shows the semantic similarity between samples derived from soft labels works better than that derived from scarce available labels. We investigate a family of regression problems in a semi-supervised setting. The task is to assign real-valued labels to a set of @math sample points, provided a small training subset of @math labeled points. A goal of semi-supervised learning is to take advantage of the (geometric) structure provided by the large number of unlabeled data when assigning labels. We consider random geometric graphs, with connection radius @math , to represent the geometry of the data set. Functionals which model the task reward the regularity of the estimator function and impose or reward the agreement with the training data. Here we consider the discrete @math -Laplacian regularization. We investigate asymptotic behavior when the number of unlabeled points increases, while the number of training points remains fixed. We uncover a delicate interplay between the regularizing nature of the functionals considered and the nonlocality inherent to the graph constructions. We rigorously obtain almost optimal ranges on the scaling of @math for the asymptotic consistency to hold. We prove that the minimizers of the discrete functionals in random setting converge uniformly to the desired continuum limit. Furthermore we discover that for the standard model used there is a restrictive upper bound on how quickly @math must converge to zero as @math . We introduce a new model which is as simple as the original model, but overcomes this restriction. Learning a classifier from private data collected by multiple parties is an important problem that has many potential applications. How can we build an accurate and differentially private global classifier by combining locally-trained classifiers from different parties, without access to any party's private data? We propose to transfer the knowledge' of the local classifier ensemble by first creating labeled data from auxiliary unlabeled data, and then train a global @math -differentially private classifier. We show that majority voting is too sensitive and therefore propose a new risk weighted by class probabilities estimated from the ensemble. Relative to a non-private solution, our private solution has a generalization error bounded by @math where @math is the number of parties. This allows strong privacy without performance loss when @math is large, such as in crowdsensing applications. We demonstrate the performance of our method with realistic tasks of activity recognition, network intrusion detection, and malicious URL detection.