Export NanoGPT Example to MPS Delegate

gkteco: <p>Following the guide here: <a href="https://pytorch.org/executorch/stable/llm/getting-started.html" rel="noopener nofollow ugc">Getting Started with LLMs via ExecuTorch — ExecuTorch 0.3 documentation</a> shows how to export nanoGPT to the XNNPACK delegate but does not provide other backend examples. The guide here <a href="https://pytorch.org/executorch/stable/build-run-mps.html" rel="noopener nofollow ugc">Building and Running ExecuTorch with MPS Backend — ExecuTorch 0.3 documentation</a> is not really what I’m looking for. I’d like a guide on how to support my own models (such as in the nanoGPT example).</p>
<p>Is there any advice on where to find this information? Or better yet, how to do this?</p>
<p>Thank you in advanced for your time.</p>

gkteco: <p>UPDATE: I’m running into the error:</p>
<pre><code class="lang-auto">CalledProcessError: Command '['flatc', '--binary', '-o', '/var/folders/fn/gg7mvyh5507g5d3gqz98r6l80000gn/T/tmpvo6v9l1i', '/var/folders/fn/gg7mvyh5507g5d3gqz98r6l80000gn/T/tmpvo6v9l1i/schema.fbs', '/var/folders/fn/gg7mvyh5507g5d3gqz98r6l80000gn/T/tmpvo6v9l1i/schema.json']' returned non-zero exit status 1.
</code></pre>
<p>when trying to export <a href="https://github.com/karpathy/nanoGPT" class="inline-onebox" rel="noopener nofollow ugc">GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs.</a> with Executorch to the MPS backend.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto"># export_nanogpt.py

import torch

from executorch.exir import EdgeCompileConfig, to_edge
from torch.nn.attention import sdpa_kernel, SDPBackend
from torch._export import capture_pre_autograd_graph
from torch.export import export
import copy

from executorch.backends.apple.mps.partition.mps_partitioner import MPSPartitioner
from executorch.exir.backend.backend_details import CompileSpec
from executorch.exir.backend.backend_api import to_backend

from model import GPT

# Load the model.
model = GPT.from_pretrained('gpt2')

# Create example inputs. This is used in the export process to provide
# hints on the expected shape of the model input.
example_inputs = (torch.randint(0, 100, (1, model.config.block_size), dtype=torch.long), )

# Trace the model, converting it to a portable intermediate representation.
# The torch.no_grad() call tells PyTorch to exclude training-specific logic.
with torch.nn.attention.sdpa_kernel([SDPBackend.MATH]), torch.no_grad():
    m = capture_pre_autograd_graph(model, example_inputs)
    traced_model = export(m, example_inputs)

# Convert the model into a runnable ExecuTorch program.
edge_config = EdgeCompileConfig(_check_ir_validity=False)
edge_manager = to_edge(traced_model,  compile_config=edge_config)

edge_copy = copy.deepcopy(edge_manager)

compiler_specs = [CompileSpec("use_fp16", bytes([True]))]
et_delegate = edge_copy.to_backend(MPSPartitioner(compiler_specs))
</code></pre>
<p>Does anyone have experience with this?</p>

kimishpatel: <p>This sounds like an issues with flatcc installed in your enviornment. Maybe <a class="mention" href="/u/dbort">@dbort</a> or <a class="mention" href="/u/larryliu0820">@larryliu0820</a> might be able to answer this more quickly.</p>

larryliu0820: <p>Hi <a class="mention" href="/u/gkteco">@gkteco</a>, if you are still running into this issue, can you please share the content of  <code>/var/folders/fn/gg7mvyh5507g5d3gqz98r6l80000gn/T/tmpvo6v9l1i/schema.json</code> from the error message? Also it would be good to share the ExecuTorch commit you are on.</p>

dbort: <p>By default, the <code>exir</code> code will delete the temp json file, but it should <a href="https://github.com/pytorch/executorch/blob/3b8b28b6915aaa367d01ceb7b7c8cd2d98f6fa67/exir/_serialize/_flatbuffer.py#L296" rel="noopener nofollow ugc">keep a copy</a> if you <code>export ET_EXIR_SAVE_FLATC_INPUTS_ON_FAILURE=1</code> before exporting.</p>
<p>Also, please post more logs if you have them. There should be log messages before the <code>CalledProcessError</code> with more details about the problem.</p>