Abstract: We examine the algebraic and geometric properties of a uni-directional GRU and word embeddings trained end-to-end on a text classification task. A hyperparameter search over word embedding dimension, GRU hidden dimension, and a linear combination of the GRU outputs is performed. We conclude that words naturally embed themselves in a Lie group and that RNNs form a nonlinear representation of the group. Appealing to these results, we propose a novel class of recurrent-like neural networks and a word embedding scheme.

Introduction: Tremendous advances in natural language processing (NLP) have been enabled by novel deep neural network architectures and word embeddings. Historically, convolutional neural network (CNN) BIBREF0 , BIBREF1 and recurrent neural network (RNN) BIBREF2 , BIBREF3 topologies have competed to provide state-of-the-art results for NLP tasks, ranging from text classification to reading comprehension. CNNs identify and aggregate patterns with increasing feature sizes, reflecting our common practice of identifying patterns, literal or idiomatic, for understanding language; they are thus adept at tasks involving key phrase identification. RNNs instead construct a representation of sentences by successively updating their understanding of the sentence as they read new words, appealing to the formally sequential and rule-based construction of language. While both networks display great efficacy at certain tasks BIBREF4 , RNNs tend to be the more versatile, have emerged as the clear victor in, e.g., language translation BIBREF5 , BIBREF6 , BIBREF7 , and are typically more capable of identifying important contextual points through attention mechanisms for, e.g., reading comprehension BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . With an interest in NLP, we thus turn to RNNs. RNNs nominally aim to solve a general problem involving sequential inputs. For various more specified tasks, specialized and constrained implementations tend to perform better BIBREF12 , BIBREF13 , BIBREF14 , BIBREF7 , BIBREF15 , BIBREF16 , BIBREF17 , BIBREF10 , BIBREF11 , BIBREF8 , BIBREF9 . Often, the improvement simply mitigates the exploding/vanishing gradient problem BIBREF18 , BIBREF19 , but, for many tasks, the improvement is more capable of generalizing the network's training for that task. Understanding better how and why certain networks excel at certain NLP tasks can lead to more performant networks, and networks that solve new problems. Advances in word embeddings have furnished the remainder of recent progress in NLP BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 . Although it is possible to train word embeddings end-to-end with the rest of a network, this is often either prohibitive due to exploding/vanishing gradients for long corpora, or results in poor embeddings for rare words BIBREF26 . Embeddings are thus typically constructed using powerful, but heuristically motivated, procedures to provide pre-trained vectors on top of which a network can be trained. As with the RNNs themselves, understanding better how and why optimal embeddings are constructed in, e.g., end-to-end training can provide the necessary insight to forge better embedding algorithms that can be deployed pre-network training. Beyond improving technologies and ensuring deep learning advances at a breakneck pace, gaining a better understanding of how these systems function is crucial for allaying public concerns surrounding the often inscrutable nature of deep neural networks. This is particularly important for RNNs, since nothing comparable to DeepDream or Lucid exists for them BIBREF27 . To these ends, the goal of this work is two fold. First, we wish to understand any emergent algebraic structure RNNs and word embeddings, trained end-to-end, may exhibit. Many algebraic structures are well understood, so any hints of structure would provide us with new perspectives from which and tools with which deep learning can be approached. Second, we wish to propose novel networks and word embedding schemes by appealing to any emergent structure, should it appear. The paper is structured as follows. Methods and experimental results comprise the bulk of the paper, so, for faster reference, § SECREF2 provides a convenient summary and intrepretation of the results, and outlines a new class of neural network and new word embedding scheme leveraging the results. § SECREF3 motivates the investigation into algebraic structures and explains the experimental setup. § SECREF4 Discusses the findings from each of the experiments. § SECREF5 interprets the results, and motivates the proposed network class and word embeddings. § SECREF6 provides closing remarks and discusses followup work, and § SECREF7 gives acknowledgments. To make a matter of notation clear going forward, we begin by referring to the space of words as INLINEFORM0 , and transition to INLINEFORM1 after analyzing the results in order to be consistent with notation in the literature on algebraic spaces.

Question: What novel class of recurrent-like networks is proposed?
Answer: A network, whose learned functions satisfy a certain equation. The  network contains RNN cells with either nested internal memories or dependencies that extend temporally beyond the immediately previous hidden state.