Bug: Building 102.0a1 on aarch64 arm64 has wasm-ld errors
Status: RESOLVED
Priority: --
Resolution: INVALID
Created: 2022-05-29 10:39:51+00:00
Resolved: 2022-05-29 10:53:21+00:00

Steps to reproduce:

Summary:

Run Amazon EC2 instance (ami-078d455b3b6ad0d4e, Ubuntu Jammy, aarch64) with the aim of building Thunderbird 91 and  Thunderbird 102, both outputting aarch64 packages.

Note: Thunderbird 91 builds OK.

Note: Thunderbird 102 has the following wasm-ld errors during the ./mach build process.  When one of these errors is fixed, the build process continues until the next wasm-ld error occurs:
1. wasm-ld: error: unable to find library -lgcc
2. wasm-ld: error: unknown file type: _muldi3.o
3. wasm-ld: error: unknown file type: letf2.o
4. wasm-ld: error: unknown file type: trunctfdf2.o
5. wasm-ld: error: undefined symbols   (8 lines of errors each naming different symbols)
6. Finally ld throws an error from rlbox.wasm.c

I am able to fix the wasm-ld errors by:
A. Forcing wasm-ld to have these default flags
     i. -L(directory containing a custom libgcc.a)
     ii. --allow-undefined
B. Removing the unknown file types from my custom libgcc.a :
    ar dv libgcc.a _muldi3.o
    ar dv libgcc.a letf2.o
    ar dv libgcc.a trunctfdf2.o
C. Deleting  line 706306 in rlbox.wasm.c    (although technically this is an ld not wasm-ld error)

The build (and the package from ./mach package) works perfectly (tested on aarch64 devices). 
I set up the system by implementing step A and step B above.  However, I need to wait until the build produces the final error (rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj') before I can delete the offending line 706306, then rerun ./mach build to a final successful build.

Detailed steps:

Note: I use llvm-13 and gcc 12.1 below.  However, I have also tried llvm-10, llvm-13, llvm-14 and llvm-15 (nightly) combined with gcc-9.3, gcc-11.2 and gcc-12.1.  The same errors occur in multiple combinations of gcc and llvm.

sudo apt update && sudo apt upgrade

sudo apt install mlocate git mercurial python3 python3-pip ansible nano watchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk \
  gcc g++ binutils \
  libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev \
  libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev \
  clang-13 clang-tools-13 clang-format-13 clangd-13 clang-tidy-13 \
  libclang-13-dev libclang-common-13-dev libclang-cpp13 libclang1-13 libclang-cpp13-dev \
  llvm-13 llvm-13-runtime libllvm13 llvm-13-dev \
  libc++1 libc++abi1 libc++-dev libgtk-3-dev libdbus-glib-1-dev \
  wget curl git gcc g++ python3 python3-pip build-essential cmake ninja-build fonts-powerline powerline \
  build-essential python3-pip cmake ninja-build python3 \
  libc++1-13 libc++abi1-13 libunwind-13 \
  lld-13 libbsd-dev emscripten \
  wasi-libc libunwind-13 libunwind-13-dev libc++abi-13-dev libc++-13-dev \
  gcc-12 g++-12 \
  shc

Install Rust 1.61 and 1.53 with:
./rustup-init.sh -y --default-toolchain stable --default-host aarch64-unknown-linux-gnu --component rustfmt

Install cbindgen 0.23.0 with:
cargo install cbindgen 

Install NodeJS 16.15.0 with nvm:
nvm install 16.15.0
nvm use 16.15.0
nvm alias default 16.15.0

Install npm 8.11.0 with:
npm install -g npm@latest

Contents of mozconfig:
# Set target and host to aarch64-unknown-linux-gnu
ac_add_options --target=aarch64-unknown-linux-gnu
ac_add_options --host=aarch64-unknown-linux-gnu
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
mk_add_options AUTOCLOBBER=1
# Disable enforcing that add-ons are signed.
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
ac_add_options --enable-official-branding
# For NSS symbols
export MOZ_DEBUG_SYMBOLS=1
# Needed to enable breakpad in application.ini
# The preceding comment appears all over the Mozilla repos, however it is misleading.
# "Official" (server) builds, as opposed to local builds, should have nothing to do
# with "breakpad" (https://chromium.googlesource.com/breakpad/) crash reporting.
# In any case, we don't want a local build.
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING=  # No telemetry.
# Used for Linux to create small launcher executable for file browsers.
# See https://hg.mozilla.org/mozilla-central/rev/3cbbfc5127e4 for details.
export MOZ_NO_PIE_COMPAT=1
mk_add_options MOZ_MAKE_FLAGS=-j16

cd ~/          ( ~/ is /home/ubuntu )
mkdir build
mkdir alternatives
cd build
hg clone https://hg.mozilla.org/mozilla-central
(Nb: latest commit was aebf5d6)
cp mozconfig mozilla-central/
cd mozilla-central
hg clone https://hg.mozilla.org/comm-central
(Nb: latest commit was aec43dd)
mv comm-central comm
cd ~/build/mozilla-central
./mach --no-interactive bootstrap --application-choice "Firefox for Desktop"
./mach build

mach build runs for 14 minutes until 'wasm-ld' is invoked and throws this error:
wasm-ld: error: unable to find library -lgcc

14:13.85 widget/gtk/mozwayland/libmozwayland.so
14:14.08 xpcom/build
14:17.13 wasm-ld-13: error: unable to find library -lgcc
14:17.13 clang: error: linker command failed with exit code 1 (use -v to see invocation)
14:17.13 gmake[4]: *** [/home/ubuntu/build/mozilla-central/config/rules.mk:498: rlbox.wasm] Error 1
14:17.13 gmake[3]: *** [/home/ubuntu/build/mozilla-central/config/recurse.mk:72: security/rlbox/target-objects] Error 2
14:17.13 gmake[3]: *** Waiting for unfinished jobs....

I couldn't figure out how to modify the source or use wasm flags to fix this so I created my own wasm-ld binary which I could switch in and out using update-alternatives.  I used shc to create compiled executable binaries from bash scripts.

Script converted to a binary:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives "$@"

When I use this custom wasm-ld with the standard gcc 12.1 libgcc.a, I get messages saying:
wasm-ld: Loading  (multiple .wasm files)
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: _muldi3.o

I modified my libgcc.a:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: letf2.o

I modified my libgcc.a further:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: trunctfdf2.o

I modified my libgcc.a further:
ar dv libgcc.a trunctfdf2.o
Note the resulting libgcc.a now had all 3 files removed.

Running ./mach build again produces:
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(algorithm.cpp.o): undefined symbol: __lttf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
 0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3

Then: I further modified my wasm-ld to allow undefined symbols.  i.e. to be equivalent to this script:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives --allow-undefined "$@"

The following 3 lines describe the final error: 

ld: /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o: in function `w2c___intscan':

/home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj'

/usr/bin/ld: libxul.so: hidden symbol `Z_envZ___multi3Z_vijjjj' isn't defined

I then deleted line 706306 in rlbox.wasm.c

Then ./mach build succeeds.