Task: Language Identification
Input (Spanish, English): The paper attempts to combine Variational Auto-Encoders with the Stick-Breaking process. The motivation is to tackle the component collapsing and have a representation with stochastic dimensionality. To demonstrate the merit of their approach, the authors test this model on MNIST and SVHN in an unsupervised and semi-supervised fashion. After reading the paper in more detail, I find that the claim that the dimensionality of the latent variable is stochastic does not seem quite correct: all latent variables are "used" (which actually enable backpropagation) but the latent variables are parametrized differently (into $\pi$) and the decoding process is altered as to give the impression of sparsity. The way all these latent variables are used does not involve any marginalization but is very similar to the common soft-gating mechanism already used in LSTM or attentional model. With respect to the Figure 5b showing the decoder input weights: component collapsing probably does not have the same effect as Gaussian prior. $\pi$ is positive therefore having a very small average value might mean that its value is close to zero most of the time, not requiring any update on the weight. For the standard Gaussian prior, component collapsing means having a very noisy input with no signal involved, which forces the decoder to shut down this channel, i.e. have small incoming weights from this collapsed variable. Adding a histogram of the latent variables in addition to that might help decide if the associated weights are relatively large because they are actually used or if it's because the inputs are zero anyway. The semi-supervised results are better than a weaker version of the model used in (Kingma et al., 2014), but as to have a fairer comparison, the results should be compared with the M1+M2 model in that paper, even if that requires also using two VAEs.
Output (English): en