explain the following paper like i am a layperson: ""Fast Inference of Mixture-of-Experts Language
Models with Offloading
Artyom Eliseev
Moscow Institute of Physics and Technology
Yandex School of Data Analysis
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Denis Mazur
Moscow Institute of Physics and Technology
Yandex
Researchcore
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Abstract
With the widespread adoption of Large Language Models (LLMs), many deep
learning practitioners are looking for strategies of running these models more
efficiently. One such strategy is to use sparse Mixture-of-Experts (MoE) — a
type of model architectures where only a fraction of model layers are active for
any given input. This property allows MoE-based language models to generate
tokens faster than their “dense” counterparts, but it also increases model size
due to having multiple “experts”. Unfortunately, this makes state-of-the-art MoE
language models difficult to run without high-end GPUs. In this work, we study
the problem of running large MoE language models on consumer hardware with
limited accelerator memory. We build upon parameter offloading algorithms and
propose a novel strategy that accelerates offloading by taking advantage of innate
properties of MoE LLMs. Using this strategy, we build can run Mixtral-8x7B with
mixed quantization on desktop hardware and free-tier Google Colab instances.
1 Introduction
Many recent advances in natural language processing rely on large pre-trained language models, such
as GPT-3 and 4 Brown et al. (2020); OpenAI (2023), Palm & Gemini Chowdhery et al. (2022); Team
et al. (2023) and many others. However, the rapid scientific progress in this area would be impossible
without open-access LLMs such as LLaMA 1 and 2 (Touvron et al., 2023), Falcon (TII UAE, 2023),
BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), or NeoX/Pythia (Biderman et al., 2023). The
key advantage of open-access LLMs is that researchers can deploy them locally and modify them in
ways that would be impossible with proprietary APIs.
Even though LLM parameters are openly available, it is still difficult to use these models due to their
sheer size. State-of-the-art open-access language models require multiple high-end GPUs 1
even for
basic inference workloads. To use these LLMs on more affordable hardware setups, one must either
compress model parameters (Dettmers et al., 2022; Frantar et al., 2022) or offload parameters to a
cheaper storage, be it RAM or SSD (Pudipeddi et al., 2020; Sheng et al., 2023).
Several recent works modify transformer architecture by introducing sparse Mixture-of-Experts
blocks (Jacobs et al., 1991; Shazeer et al., 2017). MoE blocks contain multiple “experts” (layers),
as well as a “gating function” that selects which experts are used on a given input. As a result, the
MoE block uses a small portion of all “experts” for any single forward pass, allowing for more
compute-efficient training Fedus et al. (2021); Du et al. (2022). Notably, MoEs are among the
largest Fedus et al. (2021) and among the best Mixtral AI team (2023) of available LLMs. While
Mixture-of-Experts models can be more efficient than their dense counterparts, many techniques for
efficient LLM inference were not designed with MoE in mind and perform suboptimally on modern
large language models that use mixture-of-experts layers.
1When deployed in 16-bit precision, Falcon-180B needs approximately 360GB, while LLaMA-2 70B requires
140GB of combined accelerator memory.
arXiv:2312.17238v1 [cs.LG] 28 Dec 2023
In this work, we systematically develop techniques for running large MoE language models with
limited GPU memory. Our main objective is inferencing (generating tokens) with Mixtral-8x7BInstruct — a MoE-based chat assistant — on a desktop-grade hardware where only a fraction of
experts fit into the accelerator memory. To that end:
• we observe how MoE language model accesses its experts between tokens, and find several
regularities: i) some experts are reused between adjacent tokens and ii) the model hidden
states of early layers already “know” which experts are to be used at subsequent layers.
• we design a MoE-specific offloading strategy that takes advantage of these regularities: i)
it uses LRU cache to significantly reduces GPU-RAM communication, leading to faster
generation and ii) it guesses which experts are needed ahead of time to better overlap expert
loading with computation.
• we consider the specific scenario of running Mixtral-8x7B-Instruct on a T4, RTX 3060
and RTX 3080 Mobile and develop a practical combination of mixed quantization and
the proposed offloading algorithm to run this model interactively at 2-3 tokens per second
depending on the hardware. The source code with our implementation is available online2
2 Background & Related Work
2.1 Mixture-of-Experts
The recent surge in MoE language models builds on a relatively old idea (Jacobs et al., 1991; Jordan &
Jacobs, 1994) of training ensembles of specialized models (“experts”) and a gating function to select
the right expert for the task. To achieve specialization, Mixture-of-Experts learn by simultaneously
i) training the gating function to choose the best experts and ii) training the experts themselves on
samples assigned to them by the gating function. Since then, many different MoE variants emerged,
including mixture of SVM models (Collobert et al., 2002), Dirichlet processes (Shahbaba & Neal,
2009) and various neural networks.
Shazeer et al. (2017) builds on this idea to train a sparsely gated Mixture-of-Experts to serve as
a language model. The full model consists of a recurrent neural network backbone and a MoE
module with up to 131072 experts. When processing a given token, a linear gating function select
4 most suitable experts based on the latest hidden state. The resulting model (including the gating
function and experts) is trained end-to-end to minimize cross-entropy, with an additional regularizer to
promote equal expert utilization. Shazeer et al. (2017) observed that the MoE model not only improves
perplexity, but also learns interpretable expert specializations: some experts would “specialize” on
prepositions, while others learn to express a particular concept (e.g. speed).
Since then, several lines of work explore Mixture-of-Experts with Transformer-based language
models for machine translation Lepikhin et al. (2020), masked language modeling Fedus et al.
(2021), general-purpose LLMs Du et al. (2022) and others. Most of these models follow traditional
(dense) Transformer architecture for embeddings and attention layers, and only use Mixture for the
feedforward (MLP) blocks and use a linear token-level gating function. A common observation
across most of these works is that MoE models are cheaper to train and inference Fedus et al. (2021);
Lepikhin et al. (2020), but require more parameters than a dense model with equivalent perplexity.
Pre-trained Mixture-of-Experts LLMs have been openly available for over a year3
. However, these
models seem to have gained less traction than equivalent dense models, arguable because their
sheer model size (over a trillion parameters) makes them difficult to use. Most recently, Mistral AI
released a family of sparse Mixture of Experts models called Mixtral-8x7B with near state-of-the-art
performance Mixtral AI team (2023). This model has already inspired several follow-up works and
practical applications, but it still requires a high-end GPU accelerator.
2.2 Post-training Quantization of LLMs
A natural way to circumvent this is to reduce the model size through quantization (Nagel et al., 2020;
Gholami et al., 2021; Frantar et al., 2022), sparsification Frantar & Alistarh (2023a); Ma et al. (2023),
2
https://github.com/dvmazur/mixtral-offloading
3
https://huggingface.co/google/switch-c-2048, released in November 15th, 2022
2
factorization Hsu et al. (2022), or a combination thereof. These compression types are not specific to
LLMs and are based on much older methods outside the scope of our work4
. However, recent works
found that there are unique challenges to quantizing very large transformer-based language models
due to emergent outliersDettmers et al. (2022); Lin et al. (2023); Dettmers et al. (2023).
Generally speaking, the optimal compression rate for most LLMs is 4 bits per parameter Dettmers &
Zettlemoyer (2022). While there are more extreme algorithms for 3- and even 2-bit compression Chee
et al. (2023); Lin et al. (2023); Dettmers et al. (2023), they are typically inferior to choosing a smaller
model and quantizing it to around 4 bits. Most recently, there has been several concurrent works for
quantizing Mixture-of-Experts models (Kim et al., 2023; Frantar & Alistarh, 2023b).
2.3 Inference with Parameter Offloading
A recent line of work explores inferencing and training large models with limited accelerator memory by “offloading” their parameters to another, cheaper memory, such as system RAM or even
SSD (Pudipeddi et al., 2020; Ren et al., 2021). This technique works by loading model parameters
just-in-time when they are needed for computation. Since most deep learning models use layers in a
fixed order, offloading can pre-dispatch the next layer parameters in the background, ahead of time.
This technique works particularly well when processing large batches of data, during training Pudipeddi et al. (2020); Ren et al. (2021) or large-batch non-interactive inference Aminabadi et al.
(2022); Sheng et al. (2023), where each layer processes a lot of tokens each time the layer is loaded
from RAM. In turn, when doing interactive inference (e.g. as a chat assistants), offloading works
significantly slower than on-device inference. This is because interactive inference generates tokens
autoregressively, from left to right. This way, the inference system processes one or few tokens at a
time, and therefore spends most of the time waiting for next layer’s parameters to be loaded.
2.4 Hardware Setup
While our analysis is not specific to any hardware setup, we target the hardware specifications of
cheap / free-tier cloud instances Google (2023) and the upper half of gaming computers Steam
(2023): i) enough system memory to hold model parameters, ii) a GPU with 11-16GB VRAM and iii)
host-to-device communication at 8-16GB/s (PCIe Gen.3). If we examine popular open-access MoE
models (Mixtral-8x7B and switch-c-2048), we find that all non-experts can fit a fraction of available
GPU memory. In turn, the experts that constitute vast majority of model parameters do not fit even
with quantization. Finally, even if we could fit the model parameters in memory, running generative
inference requires additional memory for layer activations and past attention keys & values.
3 Method
In this work, we aim to systematically find the optimal way to inference modern Mixture-of-Experts
LLMs on desktop or low-end cloud instances. More specifically, we focus on the task of generating
tokens interactively, i.e. generate multiple tokens per second at batch size 15
.
The generative inference workload consists of two phases: 1) encoding the input prompt and 2)
generating tokens conditioned on that prompt. The key difference between these two phases is that
prompt tokens are encoded in parallel (layer-by-layer), whereas the generation runs sequentially
(token-by-token and layer-by-layer). In general, phase 1 works relatively well with existing Mixtureof-Experts algorithms, since each layer can only be loaded once for the entire prompt. In turn, when
generating tokens, one must load layer once per each token generated. In practice, this means that
inference speed is limited by how fast one can fetch parameters from system memory.
Below, we look for patterns in how the MoE model loads its experts and propose ways to exploit
these patterns to speed up inference time.
4To learn more about these methods, please refer to surveys such as Gholami et al. (2021); Liang et al. (2021)
5As opposed to running a processing a large batch of texts over many seconds, as in Sheng et al. (2023)
3
0
1
2
3
4
5
6
Layer 0 expert #
7
Selected experts for Mixtral-8x7B-Instruct layers 0 (top) and 15 (bottom)
As
an
open
source
alternative
to
Chat G
PT
,
I
do
not
have
personal
opinions
.
However
,
I
can
provide
objective
information
about
Chat G
PT
'
s
capabilities
and
limitations
based
on
its
architecture
and
training
data
.
Chat G
PT
is
a
powerful
language
model
based
on
theG
PT
(
Gener
ative
Pre
- trained
Trans
former
0
1
2
3
4
5
6
Layer 15 expert #
7
Figure 1: An example of expert loading pattern in Mixtral-8x7B-Instruct for select layers. Blue cells
indicate that a certain expert was active when encoding a certain token; deeper blue indicates higher
gating weight. Small gray squares show which experts are cached with an LRU cache for k=2.
3.1 Expert Locality and LRU caching
As we discussed earlier in Section 2.1, Mixture-of-Experts language models were often observed to
assign individual experts to distinct sub-tasks. However, this does not mean that the model uses the
same expert over long stretches of tokens. Instead, some experts are active in short sequences of 2-4
tokens, while others are often used with “gaps”, as shown in Figure 1.
To take advantage of this pattern, we can keep active experts in GPU memory as a “cache” for
future tokens. If the same experts are activated again in future, they will be available instantaneously.
Naturally, the number of experts that can be stored this way if very limited by the available GPU
memory. For simplicity, we choose to always keep k least recently used experts as a type of LRU
cache. If k is greater than the number of active experts, the cache will save experts from multiple
previous tokens. For simplicity, we keep the same number of cached experts for each MoE layer.
We illustrate an example of how LRU cache saves experts in Figure 1 (see caption). LRU is a very
simple strategy that does not consider factors like expert activation frequencies, varying cache size
between MoE layers, or any sequential patterns in expert activation. However, we found that even
this simple strategy can significantly speed up inference for modern Mixture-of-Experts models such
as Mixtral-8x7B (see Section 4 for detailed evaluation).
3.2 Speculative Expert Loading
While LRU caching can reduce the average expert loading time, most of the inference time is still
spent waiting for the next expert to be loaded. The reason behind this is that, unlike with dense
models, MoE offloading cannot effectively overlap expert loading with computation. To understand
this problem, let us zoom into the process of generating a single token, layer-by-layer. The full
compute workload starts by embedding the previous token via look-up, then alternates between
running self-attention and MLP for each transformer block in the model. Finally, the outputs from
the last transformer block are used to predict next token logits with a linear projection.
For regular (dense) models, this architecture allows for efficient offloading schedule that pre-loads
the next transformer layer ahead of time, while the previous layer is still running. Unfortunately,
this schedule is no longer possible for Mixture-of-Experts models, where MoE MLP layers choose
which experts to load just-in-time for computation. This is because the system cannot pre-fetch
the next layer until it learns which experts should be loaded. Modern open-access MoE language
models choose active experts using the final outputs of the previous layer, which means they cannot
be pre-fetched them in parallel with previous layer.
While it is not possible6
to pre-reliably prefetch the next set of experts ahead of time, the system could
still try to guess the likely next experts and load them speculatively, while processing the previous
layer. It the guess is correct, it will speed up the next layer inference; if not, it can load the actual
next layer’s experts later. In other words, this type of speculative loading does not change the final
model predictions, but may reduce latency if the guess is accurate enough.
6More specifically, not possible without changing the model architecture, which would require re-training
4
While analyzing modern MoE models, we found that it is possible to get an accurate guess of next
layer’s experts by applying next layer’s gating function to previous layer’s hidden states — or,
more specifically, to the same hidden states that are used by previous MoE layer’s gating function.
This heuristic relies on the fact that transformer layers are residual, i.e. each layer adds to the previous
hidden states instead of re-computing them from scratch. This architecture introduces an inductive
bias such that any layer’s hidden states into a decent estimate of next layer’s hidden states.
3.3 System Design & Implementation Details
In this section, we describe practical design considerations and implementation details that we used
for inferencing MoE language models on consumer and low-end cloud hardware. Our system design
combines the caching & prefetching techniques and a mixed MoE quantization scheme .
MoE quantization. As we described earlier in Section 2.2, there are multiple weight quantization
algorithms optimized for LLMs. Model compression has a natural synergy with offloading because
compressed models take less time to load onto GPU. In our experitments, we also observed that
MoE models get better quality-size trade-offs when quantizing experts to a lower bitwidth, while
keeping all non-expert layers at 4-bit.
We use Half Quadratic Quantization (HQQ) (Badri & Shaji, 2023) — a data-free quantization
algorithm that supports a variety of bit rates. However, we chose this algorithm only for convenience,
because it was already well tested for Mixtral models. Since our analysis does not rely on any
specific choice of quantization, we believe that if we chose another quantization algorithm (e.g.
GPTQ or AWQ) our conclusions would be similar. In our early experiments, we also tried the
sub-1-bit quantization from QMoE Frantar & Alistarh (2023b) that worked well on the Switch-c-2048
model. However, we found that sub-1-bit compression caused too significant a loss in perplexity for
Mixtral-8x7B models.
Expert Offloading. As described earlier, we use LRU cache with an equal number k of cached
experts per layer. For Mixtral-8x7B, we use k=2 for 12GB GPUs and k=4 for 16GB ones. We
trigger speculative expert loading immediately after the system finished loading all experts for the
current layer. The speculative expert loading fetches 1 − 2 most likely experts. The newly loaded
experts do not replace the currently cached experts. If a speculatively loaded expert was later used
during next layer inference, it will replace the least recently used expert from the next layer’s cache.
Many consumer devices and free-tier cloud instances have limited host RAM that cannot fit the entire
model7
. In these cases, the experts must be split between host and device memory. To support this,
our implementation of expert LRU cache splits experts between host and GPU devices. When loading
and expert to the GPU cache, the system also offloads the least recently used on-device expert back
to RAM so as to preserve memory parity.
To speed up offloading in practice, we allocate all expert parameters in a contiguous memory buffer
that can be moved as a single host-to-device copy. For host-side (RAM) experts, we pin8
this
memory buffer for faster communication. Our implementation additionally allocates b=4 on-device
buffers used to copy and prefetch experts asynchronously, without modifying existing experts. These
buffers are shared between all MoE layers to reduce memory footprint. Overall, the system requires
num_layers × num_experts expert memory buffers split between host and device memory and b=4
temporary buffers, the size of each buffer being equal to a single expert.
4 Experiments
In this section, we verify our earlier hypotheses about MoE behavior and benchmark the inference
latency in different conditions. We focus our evaluations on Mixtral-8x7B and Mixtral-8x7B-Instruct
models since they represent the current state of the art among open-access MoE models. We organize
this section as follows: Section 4.1 measures the effectiveness of expert caching and pre-loading in
isolation, Section 4.2 compares different model compression algorithms and verifies our hypotheses
from Section 3.3. Finally, Section 4.3 measures the inference latency in several hardware setups.
7Notably, Google Colab RAM cannot fit Mixtral-8x7B with a reasonable compression rate
8This corresponds to tensor.pin_memory() command in PyTorch.
5
0 5 10 15 20 25 30
Layer #
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Cache hit rate
cache_size = 1
cache_size = 2
cache_size = 3
cache_size = 4
0 5 10 15 20 25 30
Layer #
0.0
0.2
0.4
0.6
0.8
1.0
Prediction recall
prefetch 1 experts
prefetch 2 experts
prefetch 3 experts
Figure 2: (left) LRU cache hit ratio for different cache size k; (right) speculative loading recall when
pre-loading a different number of experts. Regular lines represent loading 1 layer ahead; dashed line
stands for 2 layers ahead; dotted line is 10 layers ahead.
4.1 Expert LRU Cache and Speculative Loading
In this section, we benchmark the effectiveness of the two expert offloading strategies: LRU caching
and and speculative loading, as defined in Sections 3.1 and 3.2 respectively. For this evaluation, we
measure “expert recall” — the fraction of times when an expert needed for inference was already
available on GPU.
For this evaluation, we run Mixtral-8x7B-Instruct model on the OpenAssistant dataset (Köpf et al.,
2023). We test LRU caching by running the model on recorded conversations and measuring the
recall (aka “hit ratio” from caching perspective) for different cache sizes k. Next, we test speculative
loading in isolation by “guessing” which experts should be loaded (by applying the next layer’s
gating function on current layer activations), then measuring how often the actual next experts get
loaded this way. A recall of 1.0 corresponds to a situation where both (2) Mixtral active experts were
pre-fetched. We test speculative loading in three settings: 1, 2 and 10 layers ahead.
4.2 Mixed MoE Quantization
Next, we test how different Quantization schemes affect MoE performance and size. We also use
Mixtral-8x7B, but this time, we use non-instruction-tuned variant since it fits better with the available
benchmarks. We measure WikiText2 perpliexity Merity et al. (2016), C4 perplexity Raffel et al.
(2020), as well as 5-shot MMLU accuracy Hendrycks et al. (2021). Our objective for this section is
to find the best trade off between size and performance for offloading with the target setups. Note
that out of 46.7B total parameters in the Mixtral-8x7B model, the experts constitute 45.1B (96.6%).
The rest of the model parameters are allocated to embeddings, self-attention layers, MoE gates and
minor layers such as LayerNorm.
Attn
quant
Experts
quant
Model
size, GB Wiki2 C4 MMLU
FP16
FP16 86.99 3.59 6.52 70.51%
4-bit 25.82 3.67 6.58 70.3%
3-bit 23.21 3.96 6.78 69.32%
2-bit 19.33 4.52 7.31 66.66%
4-bit
FP16 85.16 3.68 6.59 —
4-bit 23.99 3.76 6.66 69.11%
3-bit 21.37 4.05 6.87 68.47%
2-bit 17.54 4.61 7.42 65.58%
Attn
quant
Experts
quant
Model
size, GB Wiki2 C4 MMLU
3-bit
FP16 85.08 3.99 6.90 —
4-bit 23.92 4.06 6.97 66.54%
3-bit 21.31 4.34 7.21 65.79%
2-bit 17.46 4.90 7.82 61.83%
2-bit
FP16 84.96 4.98 7.92 —
4-bit 23.79 5.08 8.06 59.0%
3-bit 21.18 5.36 8.34 57.67%
2-bit 17.30 5.97 9.11 55.26%
Table 1: Perplexity and model size evaluation of Mixtral-8x7B with different quantization for shared
attention (Attn quant) and experts (Experts quant) layers. For comprarison, a Mistral-7B 4-bit
quantized model has Wiki2 perplexity 5.03, C4 perplexity 7.56 and MMLU score 61.3%. See Section
4.2 for details. Green values correspond to the configurations we chose for full system evaluation.
6
Algorithm 2-bit Experts 3-bit Experts
A100 3080 Mobile 3060 T4 (Colab) A100 3080 Mobile 3060 T4 (Cloud)
Full algorithm 3.061 2.655 2.278 2.092 2.845 2.475 2.038 1.603
W/o expert pre-loading 2.918 2.227 2.051 1.567 2.683 2.024 1.857 1.365
W/o LRU cache & pre-loading 2.265 1.758 1.547 1.168 2.055 1.595 1.346 1.061
Naive offloading (accelerate) 1.392 1.059 0.919 0.661 1.246 0.914 1.791 0.580
Table 2: Inference speed for Mixtral-8x7B in low-tier , measured in tokens per second.
As discussed earlier, we use HQQ Badri & Shaji (2023) data-free quantization algorithm and consider
the following quantization schemes:
1. FP16 (no quantization)
2. HQQ 4-bit with group size 64, scale group size 256
3. HQQ 3-bit with group size 64, scale group size 128
4. HQQ 2-bit with group size 16, scale group size 128
Note that the actual model size with n-bit quantization is larger than n bits per parameter. This is
because the quantized data format also stores quantization scale and zero point for each group of
weights. Notably, the above 2-bit quantization scheme uses, on average, 2.6 bits per parameter
due to a large number of quantization schemes. We also keep embeddings, logits, MoE gates and
normalization layers in 16-bit format.
Table 1 summarizes our results: overall, it seems advantageous to quantize experts to 3 or 2 bits while
keeping attention layers to a higher bitwidth (16 or 4 bits). Based on these evaluations, we chose two
quantization schemes (highlighted in green) that offer favourable performance-size trade-offs within
the target hardware constraints.
4.3 Practical offloading performance
Finally we evaluate the performance of the Mixtral8x7B-Instruct model using the offloading techniquesproposed throughout this report. Based on the perplexity evaluations from the previous section,
we chose 4-bit HQQ quantization for the shared attention layers and 2- or 3-bit quantization for
experts. We evaluate this system by generating tokens via sampling on OpenAssistant (Köpf et al.,
2023) conversations and measuring the average number of tokens generated per second with batch
size 1. For this evaluation, we always sample proportionally to the predicted probabilities, i.e. without
temperature or nucleus sampling.
We consider four hardware configurations: a free-tier Colab instance with a T4 GPU (16GB VRAM,
PCIe Gen.3), a past generation gaming laptop with RTX 3080 Mobile (16GB, PCIe Gen.4), a midrange gaming desktop with RTX 3060 (12GB, PCIe Gen.3) and a high-end data-center server with
A100-80GB-SXM. Note that the A100 server could run the model without offloading. We use
offloading on A100 mostly to provide a reference for other setups. Finally, when evaluating 3-bit
models, we use a cloud T4 from Microsoft Azure because the free-tier colab instances did not have
enough RAM for this specific configuration. We use k = 2 for RTX 3060 and k = 4 for all other
GPUs.
As shown in Table 2, all evaluated setups can generate 2-4 tokens per second with the full algorithm.
Using pre-loading appears to be most beneficial on RTX 3060, possibly due to lower LRU cache size.
Cursiously, RTX 3060 (desktop) performs nearly equally with a much higher end 3080 Mobile. We
attribute this to the fact that both GPUs are still bottlenecked by host-to-device bandwidth, limited by
the PCIe architecture. Finally, all schemes significantly outperform naive offloading that loads the
entire MoE layer.
5 Conclusion and Future Work
In this work, we explore strategies for accelerating Mixture-of-Experts based language models on
consumer hardware with limited GPU memory. We propose a MoE-centric approach to offloading
7
and explore how mixed quantization affects perplexity and performance on language understanding
tasks. We evaluate the proposed strategies and show that they produce a significant increase in
generation speed compared to nave approaches on consumer-grade hardware, including free-tier ¨
Google Colab.
Our method provides a practical solution for inferencing large MoE language models on resourceconstricted hardware, enabling broader access to these powerful models for research and development.
As future work, we plan to explore further offloading strategies, based on speculative expert prediction""