Task: Language Identification
Input (Spanish, English): This paper gives a theoretical motivation for tieing the word embedding and output projection matrices in RNN LMs. The argument uses an augmented loss function which spreads the output probability mass among words with close word-embedding.   I see two main drawbacks from this framework: The augmented loss function has no trainable parameters and is used for only for regularization. This is not expected to give gains with large enough datasets.  The augmented loss is heavily “engineered” to produce the desired result of parameter tying. It’s not clear what happens if you try to relax it a bit, by adding parameters, or estimating y~ in a different way.   Nevertheless the argument is very interesting, and clearly written. The simulated results indeed validate the argument, and the PTB results seem promising.  Minor comments: Section 3: Can you clarify if y~ is conditioned on the t example or on the entire history. Eq. 3.5: i is enumerated over V (not |V|)
Output (English): en