Reinforcement Learning Based Asymmetrical DNN Modularization for Optimal Loading

Abstract: Latency of DNN (Deep Neural Network) based prediction is the summation of model loading latency and inference latency. Model loading latency affects the first response from the applications, whereas inference latency affects the subsequent responses. As model loading latency is directly proportional to the model size, this work aims at improving the response time of an intelligent app by reducing the loading latency. The speedup is gained by asymmetrically modularizing the given DNN model among several small child models and loading them in parallel. The decision about number of feasible child models and their corresponding split positions are taken care by reinforcement learning unit (RLU). RLU takes into account the available hardware resources on-device and provides the best splitting index k and their positions  p→ specific to the DNN model and device, where  p→=(p1,p2,...,pk) and pi  is the end position of ith  child: Mi. The proposed method has shown significant loading improvement (up to 7X) on popular DNNs, used for camera use-case. The proposed method can be used to speed up the app response. Along with that RLU driven approach facilitates for On-device personalization by separating one module only with trainable layers and loading that particular module while training on-device. 

Reviews and author responses:

The authors present a method to optimise the _loading time_ of a deep neural network on e.g. a mobile device. With ever more complicated and bigger networks (notwithstanding approaches to prune networks etc.), loading parameters can cause a noticeable initial delay before the first inference step. The authors propose an RL-based approach to optimise loading parallelism by splitting the model into smaller submodels and show that this speeds up loading time.

I've got several concerns with this submission:
- The paper was very difficult to read and follow in general, and I'm still not 100% certain my current interpretation is correct. While the language itself could use some polish, the notation is also pretty inconsistent (e.g. $S$ as state space or specific state) and the figures are hard to read. A few terms are not clearly explained: what is a block exactly? A contiguous set of weights in memory? A number of network layers? There's also mention of nodes in Section 4.2.1, are those the same? Furthermore some of the equations also seem to contain mistakes (e.g. $Block_M$ in Eq.2 should probably be $M_k$, final term in Eq. 5 should still be in the brackets).
- The problem statement does not feel like the right one to me. The problem the authors try to solve is loading delay, however the problem they are actually solving is perfect loading parallelism. Is it not more important to reduce overall delay than to make sure that each submodel can get loaded in exactly the same time? Even if the former would eventually result in the latter (which would be a nice result!), why not optimise total delay directly? Moreover, the authors allow the number of submodels to exceed the number of hardware threads, keeping the rest of them in a load buffer, which can potentially even add more delay compared to pre-specifying the number of submodels to the number of hardware threads. Also more important than the number of hardware threads, I suspect, is the number of simultaneous streams from the secondary storage medium you can effectively use due to I/O contention, which I imagine is usually lower. This would potentially also explain why in Table 1 there's only a 2.5x speed-up, where one would expect 4x with 4 submodels.
- The authors compare to an existing ("state-of-the-art") method but don't explain what this is. How does the proposed method compare to naively loading N chunks of M/N weights, with N being the number of hardware threads (or available streams from the storage medium) and M the total number of weights?
- The authors propose an RL-based approach (table-based Q-learning) to solve the problem, though provide no real motivation why such a comparatively complicated method is a good idea in this case. While RL is very flexible, it is not always the right solution.
- Though I'm still not certain, it seems like the authors use a somewhat unusual formalism in between a bandit and a full MDP, where the "action sequence" (the number and indices of the splits) is selected at the start of the episode, and then the state incrementally updated. This would explain the reward definition, which would only make sense if the number of splits $C_{split}$ would be fixed at the start of the unroll. as well as the appearance of what I interpret as the state variable $s$ in Algorithm 1. Is this interpretation correct? If so, this choice should definitely be elaborated upon in the text, as it is pretty unusual. I would also think that in this case a pure bandit-style formulation might make more sense.
- Why the choice for a sparse reward function with an $\epsilon$-margin instead of a dense reward, which are usually easier to learn?
- What is meant with "Vendor's site" and "Developer's site" in Figure 2? Are these separate geographic locations?
- While the results span different DNN models, it's unclear how the RLU controller was actually trained besides with a general Q-learning approach. How many iterations? Separate Q-tables per DNN model? What hyper parameters? Perhaps more worryingly, Table 3 contains 5 "observations". Are these simply 5 different timing measurements of the same model? Does this imply that for Table 1 only a single timing measurement was taken?

Overall I find this submission too hard to follow and agree with to vote for it to be accepted.

The authors propose a reinforcement learning approach for splitting a DNN for faster initialization. More specifically, the inference time can be split to two parts: initializing the prediction model (loading from disk, uploading to memory, initializing for inference) and the actual inference time. This paper focuses on reducing the time of the former operation.

The strength of the manuscript is that reducing the loading time has not been studied earlier. As the authors claim, most research on computational load of modern machine learning models is focused on reducing the inference time, and the loading time is dismissed.

However, the authors do not justify thoroughly why the loading time should be a concern. The startup time of the executable depends on many factors, not only the initialization of the neural network. Moreover, the loading time is a one-time event, so I fail to see the impact of whether the DNN is initialized in 0.74 seconds (proposed) versus 2.18 seconds (conventional). There might be use cases where this difference is indeed significant, but they are not presented in the paper.

The literary style of the paper is not suitable for a scientific publication. There are many errors in spelling and the clarity of presentation is poor. As a detailed note, there are no paragraph breaks, which makes the reading hard.

This paper aims at optimizing the the loading latency of DNN models by re-partition the model into modules that can be loaded in parallel. The paper adopts reinforcement learning to find the optimal partition. The experiments show that the proposed method achieved significant improvement on several popular models.

Strength of the paper: the problem this paper aims to solve is very meaningful and not addressed by previous work, as far as I know.  

Weaknesses of the paper: 
1/ This paper is not written clearly. For example, in the abstract and the beginning of the intro, the paper aims at optimizing the model loading latency, but in the later part of the intro, it also aims at optimizing on-device training? It is not clear how these two problems are closely related. 
2/ The problem formulation and methods are not expressed in clear and rigorous mathematical terms. For example, what's the meaning of equation(1) and (2)? In equation (3), what is a "split"? what is the C_split and I_split? There are verbal explanation to this, but the definition is too vague to understand. The author might want to consider representing a model as a directed-acyclic graphs (DAGs), and use the graph terminologies to describe the problem and methods.
3/ It is not clear to me what is the motivation of using RL to solve the model partition problem. Since the model is a static graph, the optimization problem can be solved by searching for an optimal partition that minizes the overall latency? The paper decomposes this process into steps, and formulate the search process as a markov decision process and use RL to search for the solution. However, it is not clear if this is necessary, or optimal. 

Overall, I think this paper is not well written and should be further polished. 


Summary:

  This paper tries to resolve the model loading issue for edge computing, which usually has limited memory such that it needs to flush out the loaded models for other applications.


Strong points:

  1. This paper points out another issue when deploying DNN models that usually be ignored. Data loading issue.

  2. This paper also deployed their method on mobile platform.

Weak points:
  1. The experimental setting is unclear and only few comparisons are available. It is hard to judge the advantage of the proposed algorithm.
  
  2. The novelty of the proposed algorithm is limited but it might be effective in practice.


Recommendation:
  Without having clear comparison, I can not evaluate the proposed algorithm.

Questions:

1. Method:

  1.1 Does inference time matters in the proposed algorithm 1? I do not see how the inference time affects the algorithm.

  1.2 How to train the model with RL specifically? In algorithm 1, does the M are a collection of models or it is the same model? My understanding is the same model and then the author would find the best split method.

  1.3 Why the number of "available" threads is not considered as a factor in deciding how to split? In the real applications, the number of "available" threads is not fixed, assuming to have access to all threads seems to be too ideal.

2. Experiments:

  2.1 it is very unclear what is the baseline (state of the art authors mentioned) authors compared, e.g. in table 1, what is the "Original" means? Does it mean load the whole model in one step? and what is "Actual" in Table 3?

  2.2 Why authors do not compare to the heuristic baseline, e.g. split the model based on the number of parameters or number of layers? This should be stronger but straightforward baselines to compare.

3. Others:

  3.1. the notation is unclear, e.g. whar is m in at the line after eq 2?

  3.2. What is s in Algorithm 1? why line 9 is not i == k?

  3.3. What is In in algorithm 2? the index i is abused, line 13 and 15 both use i.


Meta-review: The paper proposed a novel RL-based solution to the optimal partial of DNNs which is interesting to readers. 
However, the paper is not well presented and hard to follow. The lack of comparisons agains existing solutions and inconsistencies in the writing as pointed out by the reviewers largely weakens the submission. 
There's also no updates to the paper based on reviewers' comments. 

The main reason for the decision is lack of clarity and significance justifications.

Decision: withdrawn-rejected-submissions