FreeRide: Harvesting Bubbles in Pipeline Parallelism

## Abstract

Abstract. The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the main reasons for the underutilization of GPU resources in LLM training. Harvesting these bubbles for GPU side tasks can increase resource utilization and reduce training costs but comes with challenges. First, because bubbles are discontinuous with various shapes, programming side tasks becomes difficult while requiring excessive engineering effort. Second, a side task can compete with pipeline training for GPU resources and incur significant overhead. To address these challenges, we propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide provides programmers with interfaces to implement side tasks easily, manages bubbles and side tasks during pipeline training, and controls access to GPU resources by side tasks to reduce overhead. We demonstrate that FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs while serving model training, graph analytics, and image processing side tasks.

## 1. Introduction

Large language models (LLMs) are usually trained on GPUs.
As these models continue to increase in size, their GPU memory requirements can easily outstrip the capacity of a single GPU (; ).
Consequently, to accommodate this increase in size and to boost the performance of pipeline training, it is a common practice to parallelize the training of LLMs across multiple GPUs distributed over several servers.

Pipeline parallelism is a prevalent training paradigm for LLMs using multiple GPUs.
In this paradigm, the model is divided into multiple stages, each consisting of several consecutive layers.
These stages are distributed across different GPUs.
During each training epoch, a batch of input data is split into multiple micro-batches.
Each micro-batch undergoes a forward propagation (FP) and a backward propagation (BP).
The FP and BP operations on different micro-batches are carried out in parallel by the pipeline training system at each stage.
The pipeline training system schedules these operations in each epoch to train LLMs (; ; ; ; ; ; ; ; ).

An inherent limitation of pipeline parallelism is bubbles — periods in pipeline training where the GPU stays idle due to unsatisfied dependencies between FP and BP operations (; ).
Experimentally, we observe that bubbles can constitute 42.4% of the pipeline execution time, which results in significant under-utilization of GPU resources used to accelerate pipeline training.
Similar levels of under-utilization have also been reported in other studies(; ).

GPUs are crucial resources, especially those high-end models required for training LLMs (; ; ).
To enhance utilization, prior work has explored interleaving FP and BP operations (; ; ; ).
There have also been proposals to shard models into more stages and to deploy these stages on GPUs to better overlap the computation and communication (; ).
These approaches are effective for intra-epoch bubbles because they change how operations are interleaved within a pipeline epoch.
However, they do not remove the inter-epoch bubbles that occur before and after a pipeline epoch.
Prior work has also proposed to decouple the computation of gradients for the input and model weights to mitigate inter-epoch bubbles (; ).
However, they increase the size of activations, exacerbating GPU memory consumption, a common bottleneck in training LLMs.

Given the difficulty and overhead incurred in eliminating these bubbles, an alternative approach is to acknowledge their existence and utilize them by running additional workloads on a GPU.
For example, Bamboo () uses bubbles to perform redundant computation for the successive layers to improve the reliability of pipeline training on spot instances.
PipeFisher computes second-order optimization based on the Fisher information matrix to increase the convergence speed of LLM training ().
However, Bamboo and PipeFisher only target specialized procedures that are tightly coupled with pipeline training, requiring the training system and the procedures to be highly customized.
Consequently, their approaches cannot be used for generic GPU workloads.

In this paper, we present FreeRide, a holistic system to harvest bubbles in pipeline parallelism to serve extra GPU workloads as *side tasks*.
There are two main challenges that FreeRide has to overcome.
First, customizing side tasks for bubbles of various *shapes*, i.e., their duration and available GPU memory, requires enormous programming effort.
Second, LLM training requires high-end GPUs that are expensive and in high demand.
If side tasks interfere with the main pipeline training workload, e.g., accessing more GPU resources than bubbles can provide, they will slow down pipeline training and significantly increase training costs.

Our approach to overcoming the programming complexity is based on the observation that many GPU workloads naturally consist of small, repetitive steps, such as the epochs in model training that repeatedly load data and update model weights.
To reduce the programming effort, FreeRide introduces a framework that abstracts away the implementation details of side tasks, allowing programmers to adapt various side tasks to fit into the bubbles.
The key idea is to represent the life cycle of a side task, from its process creation to termination, as states in a state machine.
FreeRide provides two sets of unified interfaces — the iterative interface that features lower performance overhead, and the imperative interface that features better versatility.
They facilitate the implementation of side tasks as state transitions with little engineering effort.
FreeRide manages side tasks through these interfaces and serves them during bubbles.

FreeRide limits the GPU resource consumption of side tasks through the automated side task profiler and the side task manager.
The side task profiler first captures the key performance characteristics of the newly implemented side tasks.
The side task manager coordinates a group of side task workers, one for each GPU in the platform, and assigns each of the side tasks to one of the workers based on the characteristics.
During pipeline training, bubbles are reported to the side task manager from the FreeRide-instrumented pipeline training system.
The side task manager starts side tasks when the bubble period begins and pauses them when the bubble ends.
A side task worker deploys each side task on top of CUDA MPS () to limit its GPU memory consumption and uses a containerized environment, e.g., Docker () for isolation.
These components work collaboratively to serve side tasks during bubbles, achieving a low performance overhead on the primary pipeline training workload.

In summary, FreeRide is a holistic solution that manages and serves the side task by leveraging bubbles in pipeline training, while maintaining minimal performance overhead and requiring low programming effort.
We evaluate FreeRide by deploying it to run side tasks alongside DeepSpeed that runs pipeline training ().
We measure the time increase of pipeline training as the performance overhead caused by side tasks.
As the throughput of different side tasks is not directly comparable with the pipeline training workload, we use the cost of GPUs as a unified metric, i.e., the cost of the extra execution time from co-locating side tasks with pipeline training vs. the cost saved from running side tasks that otherwise would run on dedicated lower-tier GPUs.

The contributions of this paper are as follows:

- •
We study the bubbles in pipeline parallelism, present their various shapes in terms of duration and available GPU memory, and demonstrate their potential for side tasks.
- •
We present the programming framework and interfaces of FreeRide based on a state machine abstraction to implement generic side tasks with little engineering effort.
- •
We evaluate FreeRide with model training, graph analytics, and image processing side tasks to demonstrate FreeRide’s effectiveness in harvesting bubbles in pipeline parallelism while reducing performance overhead.
- •
By serving side tasks based on the iterative interface, FreeRide achieves average *cost savings* of 7.8% with a low performance overhead of 1.1%. This is significantly better than using CUDA MPS () directly to co-locate the tasks, which results in a 4.5% *cost increase* and 48.7% overhead. When handling a mix of these 3 types of side tasks, FreeRide achieves 10.1% cost savings with a 1.1% overhead.

## 2. Background and Motivation

In this section, we provide an overview of pipeline parallelism for training LLMs, bubbles in the pipeline, and motivation for utilizing the bubbles to execute generic workloads.

### 2.1. Pipeline Parallelism and Bubbles

Pipeline parallelism is a widely used paradigm for distributed training of LLMs that far exceed the memory capacity of a single GPU (; ; ).
In pipeline parallelism, the model is divided into multiple stages, where each stage executes several consecutive layers of the model.
These stages are deployed across different GPUs to form a pipeline.
To parallelize the computation at each stage and reduce GPU memory consumption, one batch of input data is split into micro-batches during each training epoch.
Each micro-batch undergoes forward propagation (FP) and backward propagation (BP).
In both FP and BP operations, after a stage finishes processing one micro-batch of data, it passes its output to the next stage and immediately moves on to the next micro-batch.
The FP and BP operations constitute the epochs in pipeline training systems (; ; ; ; ; ; ; ; ).
A myriad of frameworks have been developed to support pipeline training.
For example, DeepSpeed () and Megatron () are extensively used to train various LLMs such as OPT (), Turing-NLG (), and MT-NLG ().

There are periods in pipeline training when the estimated GPU streaming multiprocessor (SM) occupancy is low, as depicted by the green curves in Figure 1(a).
We refer to these periods as bubbles in the pipeline, marked as shaded areas.
Bubbles inherently exist in pipeline parallelism and occur repetitively throughout training, as they are fundamentally caused by unsatisfied dependencies between FP and BP operations (; ).
In the example of Figure 1, Stage 1 must wait for input from Stage 0 before it can execute its first FP operation, creating a bubble in Stage 1 that starts from $t+0$.

### 2.2. Bubble Characterization

To study bubbles in pipeline parallelism, we train a 3.6B-parameter LLM adapted from previous work (; ) using DeepSpeed () on a 4-GPU server (detailed setup in Section 6.1).
The training is deployed as a 4-stage pipeline, and each stage takes one GPU as shown in Figure 1.
Overall, we observe that bubbles exhibit different characteristics across all 4 stages.
Next, we take a closer look at each type of bubble.

#### 2.2.1. Bubble Categorization

We categorize the bubbles into 3 types based on their positions in the training pipeline and their causes.

$\bullet$ Type-A bubbles appear at the start and end of each epoch in all stages except for the first stage.
They are due to cascading dependencies between operations at the start and end of an epoch.
When an epoch starts, the FP operations start at Stage 0, while all other stages wait for input data from their preceding stages to start their first FP operation.
Likewise, at the end of an epoch, the last BP operation starts at Stage 3 and all other stages wait for their succeeding stages to start their last BP operation.

$\bullet$ Type-B bubbles occur in the middle of each epoch on all stages except the last one.
They are caused by dependencies between interleaved FP and BP operations.
Once the first FP operation reaches the last stage, all previous stages must wait for the corresponding BP operation before they can proceed with other operations, which causes Type-B bubbles.

$\bullet$ Type-C bubbles also occur in the middle of each epoch.
Since BP operations typically take longer than FP operations (), interleaved yet unaligned FP and BP operations create bubbles in each stage except the last.
For instance, in Figure 1(a), when Stage 2 finishes its third BP operation, it must wait for input to its fourth BP operation, which is still being processed in Stage 3, causing a type-C bubble.

Duration.
In our training setup, the duration of a bubble ranges from 0.22 to 1.04 seconds, depending on its type and stage.
The duration increases for Type-A bubbles but decreases for Type-B bubbles from Stage 0 to Stage 3.
This is because of the cascading dependency from Stage 3 to Stage 0 for Type-A bubbles and from Stage 0 to Stage 3 for Type-B bubbles.
For example, a Type-B bubble at Stage 2 is due to an unfinished BP operation at Stage 3, with the same bubble at Stage 1 caused by Stage 2.
The accumulated time to satisfy dependencies elongates Type-A or Type-B bubbles at later stages.
However, Type-C bubbles are caused by unaligned FP and BP operations.
Therefore, they have a short duration and do not exhibit the same stage-dependent variations.

Available GPU Memory.
Determined by the stage, the available GPU memory of a bubble ranges from less than 3 GB to more than 20 GB in our setup.
As shown by Figure 1(b), within a stage, the GPU memory consumption of pipeline training remains the same.
Thus, the bubbles within the same stage have the same amount of available GPU memory.
Because the later stages consume less GPU memory to store activations used by BP operations (), the available GPU memory increases from Stage 0 to Stage 3.

We further study pipeline training of models of different sizes.
As shown in Figure 2(a), bubble shapes differ.
Overall, bubbles in larger LLMs have less available memory and shorter duration, but the distributions are similar as training follows the same pipeline schedule.
Even larger models do not eliminate bubbles as they inherently exist.
Under the same configuration, the characteristics of bubbles remain the same during training as epochs are repetitive and stable.

Figure: Figure 1. A pipeline training epoch in DeepSpeed.
Refer to caption: x1.png

Figure: Figure 2. Statistics of bubbles under different model sizes.
Refer to caption: x2.png

#### 2.2.2. Bubble Rate

Besides the bubble shape, we evaluate the overall *bubble rate*, i.e., the total bubble time over pipeline training time.
When the model size increases from 1.2B to 6B parameters, as shown in Figure 2(b), both the per-epoch time in pipeline training and the total per-stage bubble time decrease.
Therefore, the bubble rate drops only slightly from 42.4% to 40.4%.
We also evaluate a larger micro-batch number, i.e., an increase from 4 (used in Figures 1 and 2) to 8.
The bubble rate drops to 26.2% as each epoch takes longer.

Prior work has focused on reducing bubbles in pipeline parallelism.
One approach is designing different ways of interleaving FP and BP operations (; ; ; ); another type of optimization divides the model into more stages and orchestrates their deployment to overlap computation and communication (; ).
These approaches optimize the scheduling strategies and interleave FP and BP operations within an epoch.
Therefore, they are effective for Type-B and Type-C bubbles that appear inside an epoch but not for Type-A bubbles.
There has also been work that reduces Type-A bubbles by decoupling the computation of gradients for the input and the model weights (; ). This comes at a cost of higher GPU memory usage due to the extra activation storage, exacerbating the GPU memory bottleneck in LLM training.
Despite these efforts, none of the approaches fully eliminate bubbles in pipeline training.

### 2.3. Utilizing Bubbles

The difficulties in mitigating bubbles in pipeline parallelism motivate an alternative approach — acknowledging their existence and leveraging their resources by allocating additional GPU workloads to them.
Prior work has utilized bubbles to run procedures that enhance pipeline training.
For example, Bamboo uses bubbles to perform redundant computation for successive layers to improve the reliability of pipeline training on spot instances (); PipeFisher computes second-order optimization based on the Fisher information matrix to speed up convergence ().
However, they tightly couple the pipeline training system with the specialized procedures.
Implementing the specialized procedures is complicated, especially since such customization should consider various bubble shapes — with durations ranging from 0.22 to 1.04 seconds, and available GPU memory from less than 3 GB to more than 20 GB on each GPU (Section 2.2).

GPUs used for training are generally compute-rich, with sufficient GPU memory available during the bubbles to accommodate other GPU workloads.
Therefore, bubbles can be used to run workloads that otherwise require dedicated GPUs.
For instance, training a ResNet18 model with batch size 64 takes only 2.63 GB of GPU memory with each iteration taking only 30.4 ms on our platform — small enough to fit into most of the bubbles in Figure 1(a).
By doing so, the resources available in bubbles present prime opportunities for serving GPU workloads, which can amortize the expensive cost of LLM training with effective GPU workload execution.
We refer to these GPU workloads served during bubbles as *side tasks*.
Prior solutions that target specialized co-running procedures (; ) do not apply to generic workloads.

In this work, we aim to *make bubble resources available to generic workloads, allowing for a programmable and efficient use of bubbles.*

### 2.4. Challenges

To execute generic GPU side tasks during bubbles, we identify two major challenges.

Challenge 1: programming effort required to implement side tasks.
Typically, GPU workloads are implemented based on the assumption that they have access to the full GPU and can run continuously until they finish execution.
However, bubbles are *intermittent* and largely vary in duration, as discussed in Section 2.2.
A side task should be tailored to bubble patterns — the side task automatically pauses or resumes when a bubble ends or starts.
Customizing the training framework to embed side tasks is conceptually feasible but limits the flexibility of implementing and executing generic GPU workloads, much like the limitations from prior work on co-running specialized procedures (; ).

Challenge 2: limiting the impact of side tasks.
LLM training can span months on expensive high-end GPUs and cost millions of dollars (; ).
Even with side tasks placed in the under-utilized bubbles, they may still interfere with pipeline training, significantly increasing the cost of LLM training and offsetting the benefit of running side tasks.
However, limiting the impact of side tasks is not trivial.
As the shape of bubbles varies, naively implementing side tasks may consume more resources than bubbles have — exceeding the duration of bubbles or even crashing the main task due to excessive GPU memory allocation.
Ideally, bubbles should be utilized without impacting the more expensive and prioritized LLM training task.

## 3. FreeRide Design Overview

FreeRide is our system that addresses the aforementioned challenges in utilizing bubbles in pipeline training to serve generic GPU side tasks.
FreeRide minimizes the performance impact of side tasks on pipeline training.
In this section, we present the design and high-level ideas of FreeRide.

### 3.1. Side Task Programming Interface

Given the high cost and priority of the main pipeline training workload,
the side task should not overlap with this main task so as to avoid competing for GPU resources.
This requirement is challenging from a programmer’s perspective as it is difficult to tailor every workload to different bubble shapes.
We observe that GPU workloads are not monolithic, rather, they can be often divided into small, repeated *steps* with largely predictable per-step duration.
For example, epochs in model training, iterations in graph analytical workloads (; ; ), and steps to process each image in image-processing workloads () all follow this pattern.
On the other hand, bubbles also demonstrate repeating and predictable patterns, as discussed in Section 2.2.

With these observations in mind, our idea is to provide an iterative programming interface that supports the step-by-step execution of side tasks to fit bubble patterns — the side tasks can be naturally divided into smaller steps to fit into bubbles of different shapes.
This interface employs a state machine abstraction for the life cycle of a side task composed of different states during its execution.
The execution of side tasks within bubbles can be implemented as state transition functions in Figure 4 (details in Section 4.1).
An execution pattern tailored to control the start and pause of side tasks through state transitions minimizes the performance impact on pipeline training.

We recognize that not all GPU workloads can be easily adapted to our iterative model.
To accommodate these workloads, FreeRide provides the imperative interface as an alternative.
The idea is to enable pausing and resuming of execution using transparent signals from a lower level.
For this reason, it does not require complex adaption but comes with a slightly higher performance overhead.
We discuss both interfaces in Section 4.2.

### 3.2. Profiling-guided Side Task Management

As bubbles have different shapes, when a side task is newly added to FreeRide, it should be assigned to a stage whose bubbles have enough GPU memory available.
When a side task is served during bubbles, there should be mechanisms that make sure the side task does not consume more resources than available by the bubbles to minimize the overhead of FreeRide, e.g., excessively allocating GPU memory or not pausing when a bubble ends.

To judiciously manage side tasks on bubbles, FreeRide leverages offline profiling to understand the shapes of bubbles.
Then, when a side task is newly submitted to FreeRide, as shown in Figure 3, FreeRide’s automated side task profiler tracks its GPU memory consumption and per-step duration.
During execution time, FreeRide employs one side task manager and multiple side task workers, one for each GPU.
The side task manager assigns the newly submitted side task to one of the side task workers which will create the side task process, based on the resulting profile.
We instrument DeepSpeed to report the start timestamps and duration of bubbles to the side task manager that will initiate state transitions of each side task through remote procedure calls (RPCs) at the start and end of each bubble.

FreeRide minimizes performance overhead on the main pipeline training workload by limiting the GPU resource consumed by side tasks (Section 4.5).
For GPU memory, the side task worker of FreeRide leverages CUDA MPS () to impose a limit on GPU memory consumed by a side task process.
For GPU execution time, FreeRide uses a twofold mechanism —
a *program-directed* mechanism through the programming interface, and a *framework-enforced* mechanism based on the side task manager and workers.
In addition, the side task worker can deploy side task processes in Docker containers () for isolation.

Figure: Figure 3. Workflow of FreeRide.
Refer to caption: x3.png

### 3.3. FreeRide Workflow

Putting the aforementioned ideas together, we present the workflow of FreeRide in Figure 3.
First, programmers adapt their side task implementation using the programming framework provided by FreeRide (step ➊).
FreeRide then automatically generates a profile of the side task’s characteristics (step ➋), which is submitted with the side task to the side task manager of FreeRide (step ➌).
After the side task is submitted for execution, based on the memory allocation of pipeline training and the characteristics of the side task, the side task manager will assign this side task to one of the workers (step ➍).
When the main pipeline training workload is running,
the side task manager continuously adds bubbles from the instrumented training framework (step ➎); at the same time, it starts/pauses side tasks based on the available bubbles (step ➏).

## 4. Implementation of FreeRide

In this section, we first introduce how FreeRide supports side tasks through its framework and interfaces.
Then, we present details of FreeRide’s profiling-guided side task management.
Finally, we discuss FreeRide’s GPU resource limit mechanisms including the implementation details.

### 4.1. Programming Framework of FreeRide

Figure: Figure 4. State transitions in a side task program.
Refer to caption: x4.png

Figure 4(a) describes the programming framework of a side task.
The framework’s core is a state machine with five states and six state transitions.
These five states capture the life cycle of a side task, from process creation to process termination, and correspond to different usage of hardware resources, e.g., GPU memory and GPU execution time.
The six state transitions are used by the programmer to implement the user-defined logic of a side task.
The programmer can override the state transition functions to define what happens during state transitions, e.g., allocating or releasing hardware resources or performing computation on GPU.
Once the side task is implemented, FreeRide automatically handles the state transitions at runtime.
Next, we discuss the states and their transitions.

$\bullet$ SUBMITTED. This state means that FreeRide has profiled a task and submitted it to the side task manager, but the side task worker has not created the side task process yet.
State transition CreateSideTask() happens automatically after the side task manager assigns a side task to a worker and the worker creates the side task process.

$\bullet$ CREATED.
In this state, the worker has created the side task process, and this process has loaded its context to the main memory but not to the GPU memory.
Take a model training side task as an example.
When it is in the CREATED state, it has already created and initialized the dataset, the data loader, the loss function, and the optimizer states in CPU memory.
However, the side task process will not load them into GPU memory until the side task manager initiates the state transition InitSideTask().
The state transition InitSideTask(), initiated by the side task manager, means that the side task will finish initialization.

$\bullet$ PAUSED.
This state is where the side task starts to use GPU memory.
The side task process has loaded its context, e.g., model weights and optimizer states, in the GPU memory.
However, this process waits in the PAUSED state until the side task manager transitions its state to RUNNING through StartSideTask().

$\bullet$ RUNNING.
In this state, the side task executes the step-wise GPU workload.
Referring to the example above of the model training side task, this step involves reading the next batch, computing the output and loss, updating the model weights, and resetting the optimizer states.
The side task iteratively enters the RunNextStep() state transition to execute these steps until the side task manager transitions its state through PauseSideTask().
Therefore, in this state, the side task process uses both the GPU memory and the GPU SMs.

$\bullet$ STOPPED.
This state marks the end of the life cycle of a side task, where the side task process releases all of its hardware resources and terminates.
It can be transited from states CREATED, PAUSED, and RUNNING through StopSideTask() initiated by the side task manager.

Figure 4(b) shows state transitions of a side task in Stage 0 of Figure 1.
Initially, the side task is in the PAUSED (P) state.
After four FP operations in the main training workload have finished, a bubble starts and the side task manager initiates StartSideTask() to transit the side task to the RUNNING (R) state.
After the first bubble ends, the side task manager pauses the side task via PauseSideTask().
Then, the main training workload has BP operations and bubbles interleaved, leading to back-and-forth transitions between PAUSED and RUNNING states of the side task.

### 4.2. Interface for Side Task Implementation

Given the FreeRide programming framework, the next step is to implement side tasks, which have two requirements.
First, the programmer should be able to implement the side task in a way that can pause at the end of a bubble and resume at the start of the next bubble.
Second, the side task should be able to communicate with the side task manager to receive state transition RPCs (Section 4.6) for pausing and resuming.
To lift programming burdens, FreeRide provides two programming interfaces supported in C++ and Python.
Once implemented using either interface, FreeRide will handle the side tasks and their state transitions transparently at runtime.
We discuss both interfaces next.

Iterative programming interface.
This is the preferred interface for side tasks in FreeRide.
It periodically checks whether the side task manager has initiated any state transitions.
If so, it executes the state transition functions in Figure 4(a) and updates the state of the side task.
Then, if the side task is currently in the RUNNING state, it executes RunNextStep().
The programmer only has to override these transition functions to implement the side task.
Pausing and resuming the side task, the transition of states, and communication with the FreeRide side task manager are all handled by the interface itself.
GPU workloads that are naturally step-wise, e.g., model training, can be easily adapted to the iterative interface.
We will discuss the adaption to the iterative interface in Section 5 using an example.

Imperative programming interface.
Not all side tasks can be explicitly implemented step-wise.
Therefore, FreeRide provides the imperative interface as a fallback solution.
The core is the function RunGpuWorkload() that allows the programmer to implement generic GPU side tasks without breaking them into steps.
When the side task manager transits the state of the side task to RUNNING for the first time, the interface calls the RunGpuWorkload() function to execute the side task.
The interface implements the pausing and resuming through signals (SIGTSTP and SIGCONT ()) and calls StartSideTask() and PauseSideTask() inside the handlers of the two signals.
The imperative interface offers better versatility at the cost of higher performance overhead (discussed in Section 5 and evaluated in Section 6.2).

### 4.3. Profiling Bubbles and Side Tasks

Bubbles.
To know the shapes of bubbles, FreeRide runs DeepSpeed, monitors its estimated SM occupancy and GPU memory consumption through the PyTorch profiler (), and automatically measures each bubble’s duration and available GPU memory.
Since the pipeline schedule determines bubbles, this offline profiling is done only once for each model and pipeline scheduling on the same hardware platform.

Side tasks.
After the programmer implements the side task, FreeRide profiles it with the automated profiling tool for its performance characteristics of GPU memory consumption and per-step duration, which FreeRide uses for side task management and GPU resource limit.
For side tasks implemented using the iterative interface, this procedure is fully automated.
The profiling tool runs the side task, monitors its GPU memory consumption, and records the timestamps at the start and end of RunNextStep() state transition for the per-step duration.
For side tasks implemented using the imperative interface, the tool profiles GPU memory consumption in the same way.
However, since the side task is not implemented step-wise, the automated profiling tool does not measure the per-step duration.

### 4.4. Side Task Management

FreeRide’s side task management has two main roles.
First, upon receiving a new side task, the side task manager assigns it to a suitable side task worker.
Second, when the pipeline training system adds bubbles to the side task manager, the side task manager initiates the state transitions of side tasks (Figure 4(a)) through RPCs.
This way, the side tasks are only served during bubbles and do not compete for GPU resources with the main pipeline training workload.

To keep track of side tasks and workers, the side task manager maintains the following fields for each worker, used by Algorithms 1 and 2 for side task management:

- •
GPUMem: the available GPU memory size.
- •
TaskQueue: the queue of side tasks ordered by submission timestamps.
- •
CurrentTask: the side task that is currently served.
- •
CurrentBubble: the bubble that is currently valid.

Figure: Algorithm 1 Procedure upon a new side task.

Algorithm 1 describes how the side task manager assigns side tasks to workers.
When the side task manager receives a new side task together with its GPU memory requirement (through profiling, Section 4.3), it first filters out all workers with enough available GPU memory (lines 2—3).
Then, from these workers, it selects the one with the smallest number of tasks (lines 4—7).
If the side task manager has selected a worker, it will assign the side task to that worker (lines 8—9).
Otherwise, it will reject the side task because of insufficient GPU memory (line 11).

Figure: Algorithm 2 Managing bubbles and side tasks.

Algorithm 2 describes how the side task manager manages bubbles and side tasks during pipeline training.
The side task manager iterates through all workers (line 2).
If CurrentBubble has just ended for a worker, the side task manager will pause CurrentTask of the worker and clear CurrentBubble (lines 3—7).
Upon a new bubble, the side task manager updates the CurrentBubble of this worker (lines 8—9).
It then checks if the worker has a CurrentTask.
If not, it will select the one with the smallest submission timestamp from TaskQueue as CurrentTask (lines 10–13).
After that,
the side task manager initiates InitSideTask() if the newly added CurrentTask is in CREATED state (lines 14—15);
otherwise, its state is PAUSED and the side task manager initiates StartSideTask() (lines 16—17).

### 4.5. GPU Resource Limit

In this section, we introduce the mechanisms in FreeRide that reduce the impact of side tasks on the main pipeline training workload through side task resource control for both GPU memory and GPU execution time.

GPU Memory.
FreeRide leverages MPS to impose GPU memory limit on side tasks, i.e., when a worker creates a side task, it sets GPU memory limits using MPS.
The side task process triggers an out-of-memory (OOM) error when its memory consumption exceeds the limit, but other processes remain unaffected.
However, FreeRide is also compatible with other mechanisms for limiting GPU memory, e.g., multi-instance GPU (MIG) () or manually implemented accounting through intercepting CUDA kernel calls ().

GPU Execution Time.
FreeRide limits GPU execution time using two mechanisms.
(1) *The program-directed mechanism* is tailored for the iterative interface.
When the side task manager makes an RPC to initiate StartSideTask() state transition of a side task, it also sends the end time of this bubble to the side task.
After the state transition finishes, the side task enters the RUNNING state.
Before the side task automatically starts RunNextStep(), the program-directed mechanism checks if the remaining time of the bubble is enough for the side task to execute the next step.
The side task will only execute the next step if the remaining time exceeds the per-step duration.
(2) *The framework-enforced mechanism* supports side tasks implemented using the imperative interface and is also a fallback mechanism for the iterative interface.
After the side task manager initiates the PauseSideTask() state transition for a side task, it waits for a short grace period before checking the last paused timestamp — a timestamp maintained by the interface that records the last time the side task was paused.
If this timestamp is not updated after the state transition begins, the side task manager assumes that the interface failed to pause the side task correctly and subsequently instructs the corresponding worker to terminate the side task process using SIGKILL.
The side task initialization, InitSideTask, which runs only once throughout the life cycle of a side task, is also protected by this mechanism.

### 4.6. Implementation

We use DeepSpeed 0.12.2 () as the framework for pipeline training.
We modify DeepSpeed in three places with 55 lines of code:
(1) before the start and at the end of an epoch for Type-A bubbles,
(2) after all FP operations preceding the first BP operation for Type-B bubbles, and (3) after the first BP operation following the last FP operation for Type-C bubbles.
The instrumented code reports bubbles to the side task manager in FreeRide, as shown in step ➎ of Figure 3.
The modifications are done once as the framework can be used for training different models.

To isolate the side task processes from the pipeline training process, FreeRide deploys workers (and side tasks of these workers) inside Docker containers, as illustrated in Figure 5.
FreeRide implements the side task manager and each side task worker in separate processes.
Communication among the pipeline training system, side tasks, and FreeRide components is facilitated through RPCs utilizing gRPC ().

Figure: Figure 5. FreeRide System Architecture.
Refer to caption: x5.png

## 5. Use of Side Tasks Interface

This section describes FreeRide’s iterative interface and imperative interface in detail.

Figure: Figure 6. Example of implementing ResNet18 training using the iterative interface of FreeRide.
Refer to caption: x6.png

Iterative programming interface.
Figure 6 is an example of implementing a side task to train ResNet18 using the iterative interface of FreeRide in Python.
Less important lines such as importing dependencies and parsing arguments are simplified.
Porting this example involves mainly 6 steps.
Step ➀: import FreeRide dependencies and inherit the iterative interface class, which includes an implementation for the state machine abstraction, communication with the side task manager, and the program-directed mechanism to limit the GPU execution time.
The programmer only has to migrate the implementation of the original GPU workload to the interface.
Steps ➁ and ➂: implement the side task initialization in 2 state transition functions, CreateSideTask() and InitSideTask(), to load the context into main memory and GPU memory respectively.
Step ➃: implement the loop condition which checks if the step-wise execution is finished.
Step ➄: implement the side task step in RunNextStep().
Step ➅: the main function handles argument parsing and runs the side task interface.

Most of the modifications are trivial, e.g., adding self. to variables such as model and optim in steps ➁, ➂ and ➄, which are required by Python.
Aside from this, the programmer can directly copy the important logic, e.g., loading the dataset and training the model, from the original implementation.
In addition, if the programmer customizes the model architecture instead of using the publicly available ones, the model implementation also does not require modification.

Imperative programming interface.
This interface does not require the programmer to implement the side task in a step-wise way.
Therefore, instead of implementing the side task in multiple functions (steps ➁ — ➄), the programmer can merge them in RunGpuWorkload(), as discussed in Section 4.2.
However, this approach trades performance for less programming effort, as pausing side tasks through the framework-enforced mechanism incurs more overheads.
When the side task manager initiates PauseSideTask() state transition via an RPC at the end of a bubble, even though the CPU process of the side task is paused by the framework-enforced mechanism (Section 4.5) after the state transition, CUDA kernels that have already started cannot be paused because they are asynchronous ().
As a result, these CUDA kernels will overlap with pipeline training, causing a higher performance overhead than the iterative interface.

## 6. Evaluation

In this section, we evaluate the benefits and overhead of using FreeRide to serve side tasks.

### 6.1. Methodology

We describe the experimental setup of our evaluation.

#### 6.1.1. Server setup.

We use a main server (Server-I) with four RTX 6000 Ada GPUs each with 48 GB of GPU memory to evaluate all pipeline training workloads and side tasks.
We use a second server (Server-II) with an RTX 3080 GPU with 10 GB of memory to run side tasks separately.
Due to the global shortage of cloud GPUs, we quote prices from a community cloud vendor () that has GPUs available.
The prices of the two servers are $P_{\rm Server-I}=\$3.96\text{/hour}$ and $P_{\rm Server-II}=\$0.18\text{/hour}$, respectively (as of June, 2024).
The price differences between higher- and lower-tier GPUs in major cloud GPU platforms are similar (; ; ).
We deploy both pipeline training and side tasks in Docker 26.1.2 ().

#### 6.1.2. Comparison points.

We evaluate FreeRide for side tasks developed with both the iterative and imperative interfaces.
For comparison, we evaluate MPS (), where we set pipeline training with the highest priority and side tasks with a lower priority.
We also evaluate a naive co-location approach by directly co-running side tasks and the main pipeline training workload on the same GPU.

#### 6.1.3. Pipeline training setup.

We train nanoGPT (; ) with model sizes 1.2B, 3.6B, and 6B with DeepSpeed 0.12.2 () in a 4-stage pipeline on Server-II (stages 0—3 in Figure 1).
We always maximize the micro-batch size (until just before OOM) to make full use of GPU memory during training.

#### 6.1.4. Side task workloads.

We implement 3 types of side tasks: model training, graph analytics, and image processing using both iterative and imperative interfaces of FreeRide.
*Model training* side tasks include ResNet18, ResNet50, and VGG19.
We use the out-of-the-box models from PyTorch () and implement the training procedure ourselves.
*Graph analytics* side tasks are adapted from Gardenia ().
It includes PageRank (PR) which is based on the PageRank algorithm () and Graph SGD (SGD) which uses stochastic gradient descent to solve matrix factorization (), both using the Orkut dataset ().
The *image processing* (Image) side task resizes an input image and adds a watermark, which we adapt from Nvidia’s example ().

#### 6.1.5. Metrics.

We use the *time increase* $\Delta{T}$ and *cost savings* $S$ in Dollars due to side tasks as metrics.
Time increase describes the performance overhead of co-locating side tasks with the main pipeline training workload.
It is defined as $\Delta{T}=(T_{\rm withSideTasks}-T_{\rm noSideTask})/T_{\rm noSideTask}$, the ratio of extra time of pipeline training with side tasks, compared with the original DeepSpeed without any side tasks.

Cost savings describe the benefits of running side tasks.
Since we cannot directly compare the throughput of different side tasks and the main pipeline training workload, we use their cost (dollars spent on GPUs) as a proxy.
First, we define the cost of pipeline training without side tasks as

$$ $C_{\rm noSideTask}=P_{\rm Server-I}\times T_{\rm noSideTask}$ $$

and the extra cost of pipeline training due to side tasks as

$$ $C_{\rm extra}=\Delta{T}\times C_{\rm noSideTask}~{}~{}~{}~{}.$ $$

Then, we compute the cost of side tasks as if each of them were executed on a dedicated lower-tier GPU as

$$ $C_{\rm sideTasks}=\sum\nolimits_{\rm Each~{}sideTask}P_{\rm Server-II}\times \frac{W_{\rm sideTask,Server-I}}{\mathit{Th}_{\rm sideTask,Server-II}}~{}~{}~{},$ $$

where $W_{\rm sideTask,Server-I}$ is the work done by a side task on Server-I, e.g., the number of epochs for model training side tasks, the number of iterations for graph analytics side tasks, and the number of images for the image processing side task.
$\mathit{Th}_{\rm sideTask,Server-II}$ is the throughput of running the same side task on Server-II, which we measure by running side tasks individually on Server-II.
Finally, we define cost savings $S$ below, where the higher the $S$ value, the greater the benefit:

$$ $S=\frac{C_{\rm sideTasks}-C_{\rm extra}}{C_{\rm noSideTask}}~{}~{}~{}.$ $$

Figure: Figure 7. Sensitivity studies of FreeRide.
Refer to caption: x7.png

**Table 1. Time increase $\Delta{T}$ (lower the better) and cost savings $S$ (positive=benefit, negative=loss, higher the better) of running DeepSpeed with side tasks.**
|  | FreeRide | Comparative Methods |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Iterative | Imperative | MPS | Naive |  |  |  |  |
| Side task | $\Delta{T}$ % | ${S}$ % | $\Delta{T}$ % | ${S}$ % | $\Delta{T}$ % | ${S}$ % | $\Delta{T}$ % | ${S}$ % |
| ResNet18 | 0.9 | 6.4 | 2.2 | 6.0 | 16.8 | -1.5 | 49.8 | -30.7 |
| ResNet50 | 0.9 | 5.3 | 3.8 | 3.9 | 19.8 | -5.1 | 61.9 | -44.0 |
| VGG19 | 0.9 | 3.9 | 5.0 | 1.4 | 21.4 | -9.1 | 53.4 | -39.7 |
| PageRank | 1.0 | 11.1 | 2.5 | 16.4 | 17.3 | 3.5 | 45.1 | -16.0 |
| Graph SGD | 1.2 | 11.8 | 4.1 | 22.8 | 231.0 | -26.7 | 62.4 | -9.1 |
| Image | 1.4 | 5.7 | 2.7 | 6.1 | 9.5 | 7.2 | 46.0 | -29.3 |
| Mixed | 1.1 | 10.1 | 4.3 | 11.0 | 24.8 | 0.2 | 64.3 | -35.5 |

### 6.2. Performance Evaluation

We run DeepSpeed to train a 3.6B model for 128 epochs with side tasks from Section 6.1.4 and compare the performance overhead, i.e., time increase ($\Delta{T}$) and cost savings ($S$) of using FreeRide with the two interfaces and the two comparative methods (as mentioned in Section 6.1.2).
For model training side tasks, we set the batch size to 64.
We run the *same side task* in all workers if they have enough GPU memory.
We also run a *mixed workload* with 4 side tasks: PageRank, ResNet18, Image, and VGG19, each in one worker corresponding to the GPU of stages 0—3 in Section 6.1.3, respectively.

The results are summarized in Table 1.
FreeRide consistently exhibits lower overhead than the comparative methods, showing only a 1.1% average time increase while achieving 7.8% average cost savings through side tasks using the iterative interface.
The imperative interface achieves comparable cost savings but with a higher overhead as it relies on the less efficient framework-enforced mechanism to limit the side task’s execution time (Section 4.5).
In comparison, the average time increase and cost savings for MPS are 48.7% and -4.5%, and for Naive are 54.7% and -29.2%.
Their *negative* cost savings indicate that these approaches can increase the total cost.
Notably, the time increase of Graph SGD with MPS is as high as 231.0%.
This anomaly is due to Graph SGD’s high compute intensity.
We conclude that FreeRide effectively utilizes bubbles in pipeline training for serving side tasks.
While the comparative methods can utilize bubbles, unlike FreeRide, they are not designed for this purpose. Thus, they are inefficient in using bubbles, leading to higher costs.

### 6.3. Sensitivity Study

We change the side task batch size, DeepSpeed model size, and DeepSpeed micro-batch numbers of different side tasks, and study the time increase and cost savings of FreeRide with the iterative interface.

(1) Varying batch sizes.
Figure 7(a) includes model training side tasks under variable batch sizes.
Other side tasks are not included as they do not run with batch sizes.
OOM means that the GPU in Server-II does not have enough GPU memory for the configuration, so the cost savings cannot be calculated.
FreeRide has low performance overheads, with around 1% increase in execution time, and cost savings of 3.4% – 7.5%.

(2) Varying model sizes.
In Figure 7(b), the performance overheads of FreeRide range from -0.7% to 1.9%, and cost savings range from 1.8% to 22.2%.
The main reason is the shorter bubble durations when training larger models as the main workload, which was also shown in Figure 2.

(3) Varying micro-batch numbers.
In Figure 7(c), the performance overhead of FreeRide increases from -0.4% to 1.5%, and cost savings reduces from 2.1% to 11.8%.
When the micro-batch number increases, because of the lower bubble rate (Section 2.2), the cost savings decrease.

Figure: Figure 8. Demonstration of GPU resource limit in FreeRide.
Refer to caption: x8.png

### 6.4. Effectiveness of GPU Resource Limit

We use training ResNet18 as an example to demonstrate the GPU resource limit mechanism in FreeRide.

Side task execution time limit.
Figure 8(a) demonstrates a case where the side task does not pause after the bubble that ends at $t+2$.
With GPU resource limit, as shown by the green and purple curves, the worker terminates the side task after a grace period via the framework-enforced mechanism.

Side task GPU memory limit.
Figure 8(b) illustrates another case where the side task keeps allocating GPU memory despite its 8 GB limit.
Without FreeRide’s GPU resource limit mechanism,
the side task’s GPU memory allocation is only capped by the physical memory limit of the GPU, potentially interfering with the main training workload.
With GPU resource limit, after the side task process exceeds its 8 GB GPU memory limit, it is terminated to release GPU memory.

### 6.5. Bubble Time Breakdown

In Figure 9, we present a breakdown of bubble utilization in FreeRide under the iterative interface.
No side task: OOM means that some bubbles are unused due to their limited available GPU memory.
For instance, the GPU memory consumption of VGG19 or the Image side task is larger than the GPU memory of bubbles in stages 0 and 1, so they cannot use half of the bubble time.
No side task: insufficient time refers to idle time because the remaining time of a bubble is not enough for the next step of the side task.
FreeRide runtime is the time consumed by running FreeRide, including the interface code and the side task manager.
Most of the bubble time with enough available GPU memory size is used by side tasks.
For side tasks with shorter per-step durations, e.g., PageRank, the ratio of FreeRide runtime is higher because more iterations of the iterative interface are executed.
In contrast, side tasks with longer per-step durations have lower bubble utilization because of insufficient time.

Figure: Figure 9. Bubble time breakdown.
Refer to caption: x9.png

## 7. Related Work

Pipeline parallelism and bubbles.
Prior research has aimed to improve the schedule of pipeline training to reduce bubbles (; ; ; ; ; ; ; ; ; ; ), and to leverage bubbles in pipeline parallelism by assigning specialized procedures coupled with pipeline training to gain benefits (; ), as discussed in Section 2.
In contrast, FreeRide does not require any changes to, or coupling with, pipeline training to serve generic GPU side tasks.

GPU sharing.
Previous work has enhanced GPU sharing through optimized GPU primitives (),
better scheduling of containers or VMs (; ),
and tailored applications like training frameworks or model compilers (; ; ; ; ; ; ).
Orion supports GPU sharing by intercepting and scheduling the CUDA kernel calls made by PyTorch().
However, Orion is agnostic of the bubbles in pipeline parallelism and can cause significant performance overhead, while FreeRide manages side tasks based on bubbles and achieves an extremely low performance overhead of about 1%.
In addition, Orion only supports deep learning workloads implemented with PyTorch while FreeRide supports generic GPU side tasks.
MPS and MIG (; ) are mechanisms provided by Nvidia for GPU sharing and virtualization.
FreeRide leverages MPS to impose GPU memory limits on side tasks.

## 8. Discussion

Security.
Prior GPU sharing solutions tend to prioritize efficiency and assume a safe environment (; ; ; ; ).
E.g., Orion assumes that the co-located GPU workloads are in the same trust domain ().
FreeRide provides the same security and isolation guarantees as the lower-level system it is built upon.
It uses MPS to limit GPU memory which provides separate GPU address spaces () for pipeline training and side tasks, and Docker for environment isolation (; ).
Orthogonally, security for co-located GPU workloads is an active research area (; ; ; ; ; ).
We expect future work to co-design security with efficient GPU sharing.

Side task management.
By implementing different strategies in its side task manager, FreeRide can incorporate more complicated management, e.g., co-locating multiple side tasks with various performance characteristics to improve the utilization of bubbles () or serving side tasks with fairness or performance guarantees (; ).

Scalability.
FreeRide can be extended for better scalability.
As FreeRide implements the communications among its components using RPCs, it can be easily extended to distributed settings with side tasks on multiple servers.
FreeRide can also be extended for multi-GPU side tasks, e.g., distributed training and big data processing (; ), by launching workers with access to multiple GPUs.

Other ML accelerators.
This work targets GPUs due to their widespread accessibility.
FreeRide’s mitigation for bubbles fundamentally applies to other ML accelerators (; ), provided that the platform has isolation and resource limit options for each process.
We anticipate future work to incorporate the approach of FreeRide with other ML platforms.

## 9. Conclusion

We propose FreeRide, a system to harvest the bubbles in pipeline parallelism to serve generic GPU side tasks.
It provides programming interfaces that abstract the life cycle of a side task as different states of a state machine and allows programmers to implement side tasks with little engineering effort.
The side task manager and side task workers manage bubbles and side tasks and reduce the performance overhead of side tasks on pipeline training.
Our evaluation shows that, on average, FreeRide achieves 7.8% cost savings for long-running, expensive, pipeline training with a negligible performance overhead of only about 1%.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: The introduction states that up to 40% of the total training time of a large language model (LLM) may be lost due to inefficiencies known as "bubbles." However, the authors also reference studies by Fan et al. (2021), Liu et al. (2023), and Qi et al. (2024), which propose techniques to minimize bubble time. After implementing these techniques, what is the average percentage of total training time that continues to be wasted on bubbles?
soundness: 3
strengths: - Original piece of work
- Experiment methodology is sound
- All components are explained well
summary: The paper presents FreeRide, a system designed to help users take advantage of idle periods in GPU task execution while training a model with pipeline parallelism. The authors contend that these idle periods, or "bubbles," cannot be completely eliminated; however, they can be utilized to make GPUs accessible for other computations during times when they are not in use.
weaknesses: The total end-to-end cost savings from running additional tasks during pipeline bubbles are relatively modest, estimated to be under 10%. In practice, this figure could be even lower unless a workload consists of small tasks with precise memory requirements that align perfectly with the available bubbles. Such gains may not be sufficient to justify the complexity of implementing FreeRide. Additionally, if the implementation is not perfect, it could result in overheads that negate any cost savings achieved. 

Given this context, it would be helpful if the authors could clarify the target users or specific products/applications for which: (1) there is a need to train a large model using pipeline parallelism while simultaneously running a smaller task that can fit within the large model's bubbles, and (2) an 8% cost savings would be significant enough to be worth considering.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why were different servers used for evaluation? A fair comparison would involve the *same* server setup to (1) train both the LLM and side tasks concurrently using FreeRide, and (2) sequentially train the LLM, then dedicate all resources to the side tasks. Comparing the total training time for (1) and (2) could demonstrate FreeRide’s effectiveness in utilizing idle resources.
2. What occurs if the primary training job crashes or finishes before the side tasks? Will the side tasks automatically adjust to use all available resources to complete remaining computations?
3. How do the experiments reflect real-world scenarios? Please provide references if some companies concurrently run LLM with ResNet or graph analytics jobs.
4. Is it possible to make LLM inference/serving as a side task? How would FreeRide support this? Such a feature might increase FreeRide's practical relevance.
5. Since pipeline parallelism is typically deployed in multi-machine settings, with 3D parallelism (i.e., data, tensor, pipeline) across machines, how does FreeRide plan to handle inter-machine communication and support 3D parallelism?
6. How does FreeRide handle checkpointing for the main and side tasks? Frequent checkpointing could increase overhead and impact idle period durations.
7. Can FreeRide support models optimized with compiler techniques (e.g., torch.compile, CUDA Graphs) during training? After leveraging the interface in Figure 5, it appears challenging to integrate other optimization methods (e.g., torch.compile, ZeRO3 parallelism). How do you plan to support these additional optimizations?
soundness: 2
strengths: 1. FreeRide introduces an effective approach for leveraging idle pipeline periods in LLM training by assigning side tasks to available GPUs.
2. The paper is well-organized and accessible, with a clear categorization of different bubble types and reasonable methods for utilizing these idle periods.
summary: This paper introduces FreeRide, a framework aimed at enhancing GPU utilization during idle periods ("bubbles") in LLM training with pipeline parallelism. FreeRide profiles workloads to capture computation and memory characteristics, then schedules side tasks on idle GPUs. Experimental results indicate that FreeRide can reduce overall training costs with minimal overhead.
weaknesses: 1. The experimental setup is questionable. Side tasks are run on an RTX3080, a GPU with lower computational capacity, whereas a fair comparison would require using the same GPU model (e.g., RTX 6000 Ada).
2. The experiments seem to be imaginary scenarios, which may not align with current industry practices. Probably utilizing bubbles for LLM inference or serving may be a more interesting use case.
3. Pipeline parallelism is primarily used in multi-machine settings, typically in combination with 3D parallelism across machines, with pipeline parallelism spanning machines instead of the single-machine setting in the paper. Communication overhead should be addressed in such settings.
4. No discussion on model checkpoints. Frequent switching or termination of side tasks could necessitate checkpointing, which incurs additional overhead.
5. The focus is heavily on system optimization, with limited ML-specific insights, raising questions about its suitability for ICLR.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: According to the iterative programming interface, side tasks and main LLM training jobs seems to be sharing memory and cuda streams directly. I would love to learn more details in isolating fatal error from side tasks, specifically, we can use illegal memory access and NCCL timeout as example fatal error to clarify

Side jobs can have side effects even they are executed successfully. One example is cuda caching allocator in pytorch. Side effects could icnrease the cache memory and increase memory fragmentation. Curious to learn more about how to overcome the side effects.
soundness: 2
strengths: * Significance: scheduling side tasks during bubbles is a generic way to improve gpu utilization. Bubbles exist not only in pipeline parallelsim but in any training jobs. One way to deal with bubble is to optimize the training itself. This paper focused on an alternative to schedule 3rd parity tasks during bubbles
* Clear programming interface: user can annotate their trainin loop with context managers at init and step, according to the the iteractive programming interface. The amount of code change is not too complicated
summary: GPUs are sometimes idle during LLM training. This paper proposed a framwork to schedule side tasks during the idle time (bubbles) to improve utilitzation. It requires careful scheduling of side tasks to avoid using too much memory or too much compute to degrade the the LLM training job itself.

From UX perspective, the paper proposed 2 approach. The one with more details are iterative programming interface. Users can annotate their training loop with context managers to create/init/step side tasks. 

From performance perspective, it's critical to understans when the bubble will happen and how long it lasts. It also critical to undersand how much memory and runtime the side task need per step. It uses pytorch profiler traces for such purpose. It also enforce maximum of gpu execution time and memory usage for side tasks.

The author breifly talked about security, fairnesss among side tasks, scalability of the framework itself.
weaknesses: * Less details in estimating bubble shapes: It briefly mentioned the usage of pytorch profiler traces. But llm training job itself can be quite dynamic. bubble shapes can change due to stragglers, variable sequence length, and multi-tenant usage (including shared usage of CPUs). It would be great if the author could explain more on how indicative the profilter traces are to predict bubbles shape for future runs
* Less details in fault tolerance: practically speaking, side tasks can be any 3rd party tasks and are error prone. It remains unclear how fatal errors from side tasks are kept self contained without affecting the main LLM training job itself. Example fatal erros are illegal memory access and NCCL timeouts.