# Essential Features in a Theory of Context for Enabling Artificial General Intelligence
## 3 Research in Context-Rich Artificial Intelligence
### Representation Learning  
Representation learning is intimately connected with the growth and success of deep learning [39; 40]. In the early days of machine learning, and even as recently as a decade ago, the primary machine learning workflow involved 'feature engineering' as a core step [41]. The importance of feature engineering was so well recognized that it was, at times, difficult to gauge whether an improvement in state-of-the-art performance on some task, such as sentiment analysis [42], was due to an algorithmic or model innovation, or due to better feature engineering [43]. Today, representation learning has largely, although not completely, eliminated the need to painstakingly engineer and evaluate sets of features. The core idea behind learning a representation of a \"data item\" (whether it is an image, a document, or even a character in a long string of text) is to \"embed\" it in a real-valued, low-dimensional vector space using a neural architecture. The argument is that such \"embeddings\" capture key properties of the data using a small set of real numbers, without requiring manual feature engineering [40]. Beyond improvements in performance that have been achieved using these methods, an impressive early finding, especially in the NLP community [44], was that such methods could capture analogies and other abstract relations that are evident in human cognition, e.g., the word2 vec model [45] could automatically complete analogies such as _King is to man as Queen is to woman_, despite never having been trained explicitly to do well on such tasks. We note that the embedding of data in low-dimensional spaces is itself not novel: previously, topic models such as Latent Dirichlet Allocation (for text) [46], and other classic dimension-reduction methods (for data that was represented more abstractly or was otherwise vectorized already) had already achieved some breakthrough innovations in this area. Today, however, representation learning is almost exclusively associated with neural representation learning. The topic receives wide coverage in the top machine learning conferences, including NeurIPS ([[https://nips.cc/](https://nips.cc/)]([https://nips.cc/](https://nips.cc/)), accessed on 1 November 2021) and the International Conference on Learning Representations (ICLR) ([[https://iclr.cc/](https://iclr.cc/)]([https://iclr.cc/](https://iclr.cc/)), accessed on 1 November 2021). In practice, representation learning is a broad area of research, and different communities approach it in different ways. In the computer vision community, the neural network itself learns the representations (given just raw inputs of pixels) that is task-optimal. The last layer of a convolutional network trained in a supervised setting, for example, could be used for getting learned representations of an image, including an unlabeled image [47]. In the NLP community, the earliest representation learning algorithms following the advent of neural networks have assumed a local definition of context. For example, models based on word2 vec (skip-gram and continuous-bag-of-words) typically slide a \"window\" of a pre-specified size over the text, and use the words in the window flanking the target word as co-context [48]. Similarly, the GloVE model uses a matrix of co-occurring words [49]. In the data mining community, researchers have shown that the premise behind word2 vec can even be applied to graphs, and have used it to learn representations for the nodes in a graph, such as a social network [50; 51], or even a geospatial network [52]. When this was first proposed in the mid-2010s, state-of-the-art performance was achieved for important data mining tasks such as link prediction, using considerably simpler systems than had been previously proposed. Gradually, the definition of context has broadened, along with (not coincidentally) the advent of more powerful and novel neural network models such as transformers, which have led to new-age embedding models such as BERT, RoBERTa and, more recently, GPT-3 [32; 34].