Before revision: We introduce sentenceMIM, a probabilistic auto-encoder for language modelling , trained with Mutual Information Machine (MIM) learning \ie . Previous attempts to learn variational auto-encoders for language data have had mixed success, with empirical performance well below state-of-the-art auto-regressive models, a key barrier being the occurrence of posterior collapsewith VAEs. The recently proposed MIM framework encourages high mutual information between observations and latent variables, and is more robust against posterior collapse.
After revision: We introduce sentenceMIM, a probabilistic auto-encoder for language modelling , trained with Mutual Information Machine (MIM) learning \ie . Previous attempts to learn variational auto-encoders for language data faced challenges due to posterior collapse. MIM learning encourages high mutual information between observations and latent variables, and is more robust against posterior collapse.