Bug: Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 16 PDT)
Status: RESOLVED
Priority: --
Resolution: WONTFIX
Created: 2022-05-29 11:07:22+00:00
Resolved: 2023-01-22 22:26:16+00:00

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Building 102.0a1 on aarch64 arm64 has wasm-ld errors

Summary:

I am running Amazon EC2 instance (ami-078d455b3b6ad0d4e, Ubuntu Jammy, aarch64) with the aim of building Thunderbird 91 and  Thunderbird 102, both outputting aarch64 packages.

Note: Thunderbird 91 builds OK.

Note: Thunderbird 102 has the following wasm-ld errors during the "./mach build" process.  When one of these errors is fixed, I could re-run the build process until the next wasm-ld error occurred:

1. wasm-ld: error: unable to find library -lgcc

2. wasm-ld: error: unknown file type: _muldi3.o

3. wasm-ld: error: unknown file type: letf2.o

4. wasm-ld: error: unknown file type: trunctfdf2.o

5. wasm-ld: error: undefined symbols   (8 lines of errors each naming different symbols)

6. Finally ld throws an error from rlbox.wasm.c

I am able to fix the wasm-ld errors by:

A. Forcing wasm-ld to have these default flags:

-L(directory containing a custom libgcc.a)
--allow-undefined

B. Removing the unknown file types from my custom libgcc.a :

ar dv libgcc.a _muldi3.o
ar dv libgcc.a letf2.o
ar dv libgcc.a trunctfdf2.o

C. Deleting  line 706306 in rlbox.wasm.c    (although technically this is an ld not wasm-ld error)

After implementing fixes (A to C), the Thunderbird 102 build succeeds.  The result from ./mach package works perfectly (tested on aarch64 devices). 

I set up the system for new builds of Thunderbird 102 by keeping my custom wasm-ld binary selected with update-alternatives and making it use a custom libgcc.a

However, I need to wait until the Thunderbird 102 build produces the final error (rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj') before I can delete the offending line 706306.  Finally, I rerun ./mach build to a successful build.

Detailed steps:

Note: I use llvm-13 and gcc 12.1 below.  However, I have also tried llvm-10, llvm-13, llvm-14 and llvm-15 (nightly) combined with gcc-9.3, gcc-11.2 and gcc-12.1.  The same errors occur in multiple combinations of gcc and llvm.

sudo apt update && sudo apt upgrade

sudo apt install mlocate git mercurial python3 python3-pip ansible nano watchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk gcc g++ binutils libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev clang-13 clang-tools-13 clang-format-13 clangd-13 clang-tidy-13 libclang-13-dev libclang-common-13-dev libclang-cpp13 libclang1-13 libclang-cpp13-dev llvm-13 llvm-13-runtime libllvm13 llvm-13-dev libc++1 libc++abi1 libc++-dev libgtk-3-dev libdbus-glib-1-dev wget curl git gcc g++ python3 python3-pip build-essential cmake ninja-build fonts-powerline powerline build-essential python3-pip cmake ninja-build python3 libc++1-13 libc++abi1-13 libunwind-13 lld-13 libbsd-dev emscripten wasi-libc libunwind-13 libunwind-13-dev libc++abi-13-dev libc++-13-dev gcc-12 g++-12 shc

Install Rust 1.61 and 1.53 with:
./rustup-init.sh -y --default-toolchain stable --default-host aarch64-unknown-linux-gnu --component rustfmt

Install cbindgen 0.23.0 with:
cargo install cbindgen 

Install NodeJS 16.15.0 with nvm:
nvm install 16.15.0
nvm use 16.15.0
nvm alias default 16.15.0

Install npm 8.11.0 with:
npm install -g npm@latest

Contents of mozconfig:

ac_add_options --target=aarch64-unknown-linux-gnu
ac_add_options --host=aarch64-unknown-linux-gnu
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
mk_add_options AUTOCLOBBER=1
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
ac_add_options --enable-official-branding
export MOZ_DEBUG_SYMBOLS=1
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING=  # No telemetry.
export MOZ_NO_PIE_COMPAT=1
mk_add_options MOZ_MAKE_FLAGS=-j16

cd ~/          ( ~/ is /home/ubuntu )
mkdir build
mkdir alternatives
cd build
hg clone https://hg.mozilla.org/mozilla-central
(Nb: latest commit was aebf5d6)
cp mozconfig mozilla-central/
cd mozilla-central
hg clone https://hg.mozilla.org/comm-central
(Nb: latest commit was aec43dd)
mv comm-central comm
cd ~/build/mozilla-central
./mach --no-interactive bootstrap --application-choice "Firefox for Desktop"
./mach build

mach build runs for 14 minutes until 'wasm-ld' is invoked and throws this error:
wasm-ld: error: unable to find library -lgcc

14:13.85 widget/gtk/mozwayland/libmozwayland.so
14:14.08 xpcom/build
14:17.13 wasm-ld-13: error: unable to find library -lgcc
14:17.13 clang: error: linker command failed with exit code 1 (use -v to see invocation)
14:17.13 gmake[4]: *** [/home/ubuntu/build/mozilla-central/config/rules.mk:498: rlbox.wasm] Error 1
14:17.13 gmake[3]: *** [/home/ubuntu/build/mozilla-central/config/recurse.mk:72: security/rlbox/target-objects] Error 2
14:17.13 gmake[3]: *** Waiting for unfinished jobs....

I couldn't figure out how to modify the source or use wasm flags to fix this so I created my own wasm-ld binary which I could switch in and out using update-alternatives.  I used shc to create compiled executable binaries from bash scripts.

Script converted to a binary:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives "$@"

When I use this custom wasm-ld with the standard gcc 12.1 libgcc.a, I get messages saying:
wasm-ld: Loading  (multiple .wasm files)
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: _muldi3.o

I modified my libgcc.a:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: letf2.o

I modified my libgcc.a further:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: trunctfdf2.o

I modified my libgcc.a further:
ar dv libgcc.a trunctfdf2.o
Note the resulting libgcc.a now had all 3 files removed.

Running ./mach build again produces:
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(algorithm.cpp.o): undefined symbol: __lttf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3

Then: I further modified my wasm-ld to allow undefined symbols.  i.e. to be equivalent to this script:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives --allow-undefined "$@"

The following 3 lines describe the final error: 

ld: /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o: in function `w2c___intscan':

/home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj'

/usr/bin/ld: libxul.so: hidden symbol `Z_envZ___multi3Z_vijjjj' isn't defined

I then deleted line 706306 in rlbox.wasm.c

Then ./mach build succeeds.