Change:
<!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --I tried this code: for a file that is <1MB large. I expected to see this happen: The read should allocate a vec that is as large as the file content. The heap size in Emscripten should not meaningfully change. Instead, this happened: The Emscripten heap grows by 1.8GB I looked into the implementation of fs::read. Since it uses the file metadata size to pre-allocate the output vector, I debug printed the file size, which is around 1.8GB. Since my Rust code is running inside Pyodide 0.25, I also checked Python's of the file, which reports the correct file size. Reading the file in Python also succeeds without the enormous over-allocation. It seems that this is a Rust-specific bug in getting the correct file size information from Emscripten. <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --My Rust toolchain is pinned to nightly-2024-07-21, the last nightly for 1.81.
uhh does this reproduce on the latest nightly?
Yes, unfortunately it still reproduces on . My current workaround is the following:
It seems we just call fstat64? It's not clear how that's not working out for us.
We could have an incorrect definition of the Emscripten stat64 structure?
emscripten's definition of changed in version 3.1.42, which . Maybe that can lead to version-mismatch problems? Which version are you using? From , it seems like the Rust standard library still targets 2.0.5.
Pyodide v0.25 uses Emscripten 3.1.46, so that would be after the change. The fix to libc was in and released in v0.2.148, which Rust is already using. Huh.
When building with a fixed version of , it does a check in the build script to see if the installed version of emscripten uses the old or new definition. I assume you downloaded the emscripten target from rustup, which would mean that you're using a precompiled standard library. So the question is really which version of emscripten is installed on the builders that built the standard library you are using.
Do you think that using would help here?
Sure, give it a try!
Yes, works both on the latest nightly and my pin. How could this footgun be avoided?
Not sure; maybe the emscripten version used in could be updated and then a minimum version requirement for the emscripten toolchain could be . Note that I'm not a target maintainer or anything, just a fellow contributor :)
Perhaps we could add some warning to the target list about this issue? Could someone point me in the direction of which file to edit? Cc Pyodide packages using Rust should be built with build-std to ensure that they use the correct Emscripten ABI, otherwise e.g. file size metadata is garbage
Here, you would need to add an entry because the emscripten target doesn't have a doc for it yet, but you can add appropriate caveats to this page and in this folder:
It seems you needed to also update to finish the patch
Ah, I think this is also the reason why failed without . I just opened (draft) PR for this in libc to bump emsdk and make the changes in unconditional.
Thanks for reporting this! I will add a test for this to Pyodide. Yeah does sound like we should add back in to the rust flags. It's definitely not safe to build against one version of Emscripten and then link against a different one. I thought we'd been passing it all along, but looking through Pyodide's git history I got rid of it almost immediately after adding rust support. We should probably encourage everyone targeting Emscripten to pass . Not doing that requires ABI stability. Seems like I got rid of because I was hoping to get off of nightly, but we're stuck on nightly anyways because we have to pass because Emscripten fails if you pass when linking a shared library. It would be really nice if Emscripten could adopt an abi versioning scheme so that other projects could know what is going on.
By the way, is it possible for us to build the rust standard library once against our emscripten compiler and point all packages we build to this one copy of the rust stdlib? That way we would only have to rebuild when we change the version of rust or emscripten as opposed to every time we build any rust package. It would save a lot of CI time.
When building the Rust crates, you could pass to a prebuilt sysroot. To each crate, it would look again just normal (no anymore). I'm not sure how to prepare the sysroot itself, but there is a crate that supports building with custom sysroots, so it might be useful to look at.
WG-prioritization assigning priority (). label -I-prioritize +P-high

Discussion:
git clone https://github.com/emscripten-core/emsdk.git /emsdk-portable cd /emsdk-portable <del> hide_output ./emsdk install 2.0.5 ./emsdk activate 2.0.5 </del> <ins> hide_output ./emsdk install 3.1.68 ./emsdk activate 3.1.68 </ins>