Efficient Methods in Deep Learning Lifecycle: Representation, Prediction and Model Compression
Creators: name: Sha, Long
Publisher: name: Brandeis University
Year: 2021
Subjects: deep learning
model compression
representation learning
temporal learning

description: The proliferation of digital technologies has led to an explosion in the number of large datasets available in the last few years, placing traditional machine learning approaches to data processing and modeling at a competitive disadvantage. Nevertheless, analyzing complex, high-dimensional, and noisy datasets can be a tremendous challenge. Deep learning, as part of a broader family of machine learning methods, has shown superior performance in dealing with such challenges in the past decade.
description: However, several challenges in the deep learning lifecycle hinder the performance and democratization of deep learning methods. This dissertation spotlights a key challenge: efficiency. Specifically, we focused on three topics: efficient representation learning, efficient temporal model learning, and efficient model compression. The three topics correspond to the sequential stages of data representation, modeling, and deployment in the deep learning lifecycle.
description: The first topic is efficient representation learning. Our research focuses on the field of knowledge graph representation learning. Though researchers have investigated several knowledge graph embedding methods, efficiently comparing them with existing solutions and exploring new ones remains challenging. We have, thus, proposed a unified group-theoretic framework for general knowledge graph embedding problems and explored two novel efficient embedding methods that though compact in size, demonstrate impressive results on benchmark datasets.
description: The second topic is efficient temporal model learning. As a significant part of artificial intelligence, temporal learning utilizes temporal data to predict future events or infer latent traits. We found that numerous deep learning methods are focused on computer vision and natural language processing though efficient prediction models for temporal learning are in demand. This thesis proposes three efficient prediction models in temporal learning that can deliver superior performance while providing interpretable insights into the model and the task. The first model pertains to efficient knowledge tracing, which analyzes students' learning activities to attempt to quantify how well they master the knowledge components. The second model is for studying the epidemic data of the novel coronavirus SARS-CoV-2 to predict trends and examine the impact of its environmental factors. The third model utilizes longitudinal electronic medical records to predict patient mortality risk; this can help identify high-risk patients.
description: The third topic is efficient model compression. We found that most state-of-the-art deep learning methods typically require substantial memory and storage overhead. This hinders their use of edge computing devices. Deep learning models with a large number of parameters and great computational complexity also consume significant energy, making their deployment on battery-limited devices difficult. To tackle this challenge, we have proposed a probabilistic inference method for pruning deep neural networks, which efficiently compresses the model and ensures minimum performance loss.
description: In this dissertation, we have discussed efficient methods in the deep learning lifecycle, especially in representation learning, prediction, and model compression. We hope that our contributions serve as a catalyst for deep learning democratization and inspire further exploration of the subject.