Repository: spack/spack

Issue:
support aarch64 for go-bootstrap/go
I found that go-bootstrap/go package cannot support aarch64 now.

If we want to support aarch64 , the flow is like that:

> 1)  Download the 1'st arm64 version as below:
> https://dl.google.com/go/go1.8.7.linux-arm64.tar.gz
>
> 2) unpack `go1.8.7.linux-arm64.tar.gz`
>
> 3) export `GO_BOOTSTRAP=/xxx/go` 
>
> 4) After finishing above steps, the latest go package can be download and compile.

But the question is that: I think it's totally deferent way of x86 bootstrap compare for the above steaps.
So I'm not sure how to modify the go-bootstrap package.py.

Is there any suggestion for developing this patch?


Discussion:
@hartzell 
Yeah, the Go compiler is written in Go, so you need a bootstrap compiler to build it.  The last Go compiler written in C was the 1.4 release, and the Go team is committed to keeping it alive for people who insist on bootstrapping from a C environment.

But `aarch64` support didn't arrive until 1.5 (according to the [intro]), and @darmac's comment suggests that one needs 1.8.7.

[The docs][bootstrap] list 4 options for getting a bootstrap compiler:

1. Bootstrap toolchain from binary release 
2. Bootstrap toolchain from source
3. Bootstrap toolchain from cross-compiled source
4. Bootstrap toolchain using gccgo

As mentioned above, 2 is out.  I can't see anything useful to be done with 3 either.

That leaves 1 and 4 and leads to a Philosophical Question about how Spack ought to do things:  is it worth the trouble to build an `aarm64` gcc with go support so that we continue to always build things from source *or* should we grab one of the Go team's builds and use it the same way we use the system C compiler?  How important is the build-all-the-things-from-source ethos?

I'm probably not the best arbitrar for that question, I don't always seem to look at things the same way as other Spack users.  Hopefully others will chime in.  @adamjstewart -- thoughts on who to tag?

I don't have access to an `aarch64` system.  @darmac -- can you play around with using Spack to build `gcc@<VERSION_TBD> languages=go` and report back about how long it takes and how tricksy it is?  I'm unsure about the version, the [bootstrap] docs seem to suggest something in the 5 series will work, but...

If we go with 1, then some package needs to fetch the tarball, unpack it some place, and setup any additional necessary environment variables to use it.  This could be conditional logic in the `go-bootstrap` package or a separate `go-binary-bootstrap`.  If we go with the latter, the `go` package would need to `depend_on` the correct thing as the situation requires.

If we go with 4, then it could either be hidden in the `go-bootstrap` package or the `go` package could be be adjusted to conditionally `depend_on` `gcc@VERSION languages=go`, where VERSION is tbd.

Either way, perhaps *all* of the go architectures should switch to using method 1 or 4.  That'd be a big bunch of churn, but the consistency might be worth it.   Again, feedback....

[intro]: https://golang.org/doc/install/source#introduction
[bootstrap]: https://golang.org/doc/install/source#go14
I’m personally fine with grabbing a Go binary and using it to bootstrap a newer version. We use binaries for other packages like Java. And since were using it to bootstrap, the final product is still built from source and optimized.
Yeah, it's nicely analogous to how the icedtea package depends on jdk; the jdk package just unpacks stuff, install_tree's it into place, and does the appropriate setup_dependent_* stuff.

So, that's option 1.
Do we need other inputs?  I've guessed wrong a few times lately....

What about reworking the dependency for *all* the platforms?
@hartzell As far as I'm concerned, you're in charge of anything Go-related in Spack. If you know other Spack users who use Go, you can ping them, but I'm fine with letting you make the final call. I agree with doing things uniformly across _all_ platforms.
Ok, pinging a couple of other folks who've had their fingers in the package at one point or another and might have constructive input: @trws @paulbry @michaelkuhn @matthiasdiener 
@darmac -- What machines are you using?  [AWS Graviton/A1](https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/) instances or ???
While I originally wrote it, I haven't touched this in some time.  For whatever it's worth, the plan at the time was to support this kind of thing by at some point allowing gccgo to provide the dependence through a virtual dep or a compiler dep (if those ever materialized).

It would probably be most general to find a way to allow gccgo to provide this, but if we're stuck with a binary it's not the end of the world.  I'd prefer it be scoped as narrowly as possible though, there are too many binary packages that blindly grab a single version regardless of actual architecture compatibility around.
@trws -- Thanks for the feedback.  I'm having parsing this sentence:

> I'd prefer it be scoped as narrowly as possible though, there are too many binary packages that blindly grab a single version regardless of actual architecture compatibility around.

Which scoping?  Are you suggesting that we only use binary bootstrapping for packages that we *can't* build from source, and go@1.4 for the rest?

One interesting way to figure out what binary package to use to bootstrap `go@<VERSION>` would be to just use the Go team provided relase of Go at `<VERSION>`.  That should ensure that no features are in use that an older binary doesn't support.
Yup, that's precisely what I meant.  Use it for only the architectures or similar that are absolutely required, or at least only those where it's verified to work.  There are too many ways for the binary packages to go wrong unless they're only enabled for exactly what they support, specific architecture, operating system, calling convention, perhaps not here but usually also libc version, etc.
> @darmac -- What machines are you using? [AWS Graviton/A1](https://aws.amazon.com/blogs/aws/new-ec2-instances-a1-powered-by-arm-based-aws-graviton-processors/) instances or ???

=> I'm using [TaiShan2280-V2](https://e.huawei.com/en/products/servers/taishan-server/taishan-2280-v2) as test machines. And the CPU is based ARMv8 (Kunpeng920 and Kunpeng916).

> @darmac -- can you play around with using Spack to build `gcc@<VERSION_TBD> languages=go` and report back about how long it takes and how tricksy it is?

=> I will tried it and feedback the result.

> That leaves 1 and 4 and leads to a Philosophical Question about how Spack ought to do things

=> Just in my opinion, I think option 1 is much easier to transport the go-bootstrap and go packages to aarch64 platform. I seldom use the go language, so I'm not sure if option 4 is a better way, what I tried is compile latest go1.13.7 by using go1.8.x. And also, if we can run pass the `GO Unit Test Cases`, the compatibility risk will be reduced?

Anyway, thanks so much for your suggestion.
@hartzell 

> @darmac -- can you play around with using Spack to build gcc@<VERSION_TBD> languages=go and report back about how long it takes and how tricksy it is?

=> I checked it, and can build it automatically. Then I tested a `hello_world.go` and run it well.

I have tried both way of Bootstrap toolchain manually, both of them can work well.

**Method 1: Bootstrap toolchain using gccgo**
`spack spack install -v gcc languages=go`
`spack load gcc languages=go`

**Method 2: Bootstrap toolchain from binary release**
`wget <go_src_tarball>`
`export GOROOT=<your path>`
`export PATH=$GOROOT/bin:$PATH`

**GO TOOL BUILD** : 
After bootstrap is ready for using, we can start build go tools
`wget <go_src_tarball>`
`cd go/src/`
`./make.bash`

I think these steps above is quite easy, and it should work in most of platforms.
So, do you have any suggeustion for the next?
Shall we refine the go-bootstrap package?
@hartzell  @adamjstewart 
Is there any suggestion update?
Hey all, just ran into this attempting to compile Go on an Arm based Mac. If bootstrapping with gccgo is possible. Would it be feasible for the go-bootstrap package to just be a wrapper around installing gcc with the go language support enabled? 

Are there any specific use cases where compiling GCC wouldn't be an option and we would have to fallback to a binary bootstrap?
Not that I know of. That’s a good solution, but an expensive one if you don’t have gccgo available already. That’s why we started the way we did, even bootstrapping from an ancient one through c was a great deal faster, but that should be much more reliable if it actually covers all the versions and platforms.

Sent from my iPhone

On May 6, 2021, at 11:22 AM, Alec Scott ***@***.***> wrote:

﻿

Hey all, just ran into this attempting to compile Go on an Arm based Mac. If bootstrapping with gccgo is possible. Would it be feasible for the go-bootstrap package to just be a wrapper around installing gcc with the go language support enabled?

Are there any specific use cases where compiling GCC wouldn't be an option and we would have to fallback to a binary bootstrap?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https://github.com/spack/spack/issues/14900*issuecomment-833754747__;Iw!!G2kpM7uM-TzIFchu!iKD5ehsMV0nmp_OLsrKGVAXbGp3yx1iK4vMKxPG1aS8aAT5_uymp_stiA5BWL4M3Wg$>, or unsubscribe<https://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAFBFNJTM77LRUJH2NE6ASLTMLM5VANCNFSM4KTRBDBQ__;!!G2kpM7uM-TzIFchu!iKD5ehsMV0nmp_OLsrKGVAXbGp3yx1iK4vMKxPG1aS8aAT5_uymp_stiA5A5TCp0KA$>.

I'd like to point out that the currently, AArch64 users can't actually build Go with Spack.  Pull #28850 changed go-bootstrap/package.py to conflict when the target is aarch64 and the platform is darwin and changed also go/package.py to remove the option to bootstrap using gccgo.  Unfortunately, as discussed above, the go-bootstrap is too old to support **any** AArch64 hardware, so go-bootstrap doesn't work even if you're not using Apple hardware.

I've updated both package.py files locally so that I can build go using gccgo on our AArch64 cluster.  Should I create a pull request for these changes? 

Also note that there's a bug in some versions of gcc that causes the build of go to fail so you have to specify particular versions of gcc in the `depends_on` line in go/package.py.   (See golang/go#47771.)
If it works, yes you definitely should.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/go-bootstrap/package.py b/var/spack/repos/builtin/packages/go-bootstrap/package.py
--- a/var/spack/repos/builtin/packages/go-bootstrap/package.py
+++ b/var/spack/repos/builtin/packages/go-bootstrap/package.py
@@ -36,8 +36,8 @@ class GoBootstrap(Package):
     depends_on('git', type=('build', 'link', 'run'))
 
     conflicts('os=monterey', msg="go-bootstrap won't build on new macOS")
-    conflicts('target=aarch64:', when='platform=darwin',
-              msg='Go bootstrap is too old for Apple Silicon')
+    conflicts('target=aarch64:',
+              msg="Go bootstrap doesn't support aarch64 architectures")
 
     def patch(self):
         if self.spec.satisfies('@:1.4.3'):
diff --git a/var/spack/repos/builtin/packages/go/package.py b/var/spack/repos/builtin/packages/go/package.py
--- a/var/spack/repos/builtin/packages/go/package.py
+++ b/var/spack/repos/builtin/packages/go/package.py
@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 import os
+import platform
 import re
 
 import llnl.util.tty as tty
@@ -130,7 +131,20 @@ class Go(Package):
     provides('golang')
 
     depends_on('git', type=('build', 'link', 'run'))
-    depends_on('go-bootstrap', type='build')
+
+    # aarch64 machines (including Macs with Apple silicon) can't use
+    # go-bootstrap because it pre-dates aarch64 support in Go.  These machines
+    # have to rely on Go support in gcc (which may require compiling a version
+    # of gcc with Go support just to satisfy this requirement).  However,
+    # there's also a bug in some versions of GCC's Go front-end that prevents
+    # these versions from properly bootstrapping Go.  (See issue #47771
+    # https://github.com/golang/go/issues/47771 )  On the 10.x branch, we need
+    # at least 10.4.  On the 11.x branch, we need at least 11.3.
+
+    if platform.machine() == 'aarch64':
+        depends_on('gcc@10.4.0:10,11.3.0: languages=go', type='build')
+    else:
+        depends_on('go-bootstrap', type='build')
 
     # https://github.com/golang/go/issues/17545
     patch('time_test.patch', when='@1.6.4:1.7.4')


Test patch: