Repository: tetratelabs/wazero
Pull request #773 (2022-08-29 01:40:57)

site: Adds content to languages page
This consolidates copy/pasted notes into the main languages page, and
elaborates more general concerns than existed before.


codefromthecrypt: https://deploy-preview-773--wazero.netlify.app/

Patch:
diff --git a/internal/compilationcache/file_cache.go b/internal/compilationcache/file_cache.go
index fc7e0116..6811d886 100644
--- a/internal/compilationcache/file_cache.go
+++ b/internal/compilationcache/file_cache.go
@@ -7,6 +7,7 @@ import (
 	"io"
 	"os"
 	"path"
+	"runtime"
 	"sync"
 )
 
@@ -40,7 +41,7 @@ type fileReadCloser struct {
 }
 
 func (fc *fileCache) path(key Key) string {
-	return path.Join(fc.dirPath, hex.EncodeToString(key[:]))
+	return path.Join(fc.dirPath, runtime.GOARCH+"-"+runtime.GOOS+"-"+hex.EncodeToString(key[:]))
 }
 
 func (fc *fileCache) Get(key Key) (content io.ReadCloser, ok bool, err error) {
diff --git a/site/content/languages/_index.md b/site/content/languages/_index.md
index 17edd01d..47bd2662 100644
--- a/site/content/languages/_index.md
+++ b/site/content/languages/_index.md
@@ -29,24 +29,91 @@ maintain language compilers. If you see any errors, please help [maintain][1]
 these and [star our GitHub repository][2] if they are helpful. Together, we can
 make WebAssembly easier on the next person.
 
+## Constraints
+
+The [WebAssembly Core specification]({{< ref "/specs#core" >}}) defines a
+stack-based virtual machine. The only features that work by default are
+computational in nature, and the only way to communicate is via functions,
+memory or global variables.
+
+WebAssembly has no standard library or system call interface to implement
+features the operating system would otherwise provide. Certain capabilities,
+such as forking a process, will not work. Support of common I/O features, such
+as writing to the console, vary. See [System Calls](#system-calls) for more.
+
+Software is more than technical constraints. WebAssembly remains a relatively
+niche target, with limited maintenance and development. This means that certain
+features may not work, yet, even if they could technically.
+
+In general, developing with WebAssembly is difficult, and fewer problems can
+be discovered at compilation time vs more supported targets. This results in
+more runtime errors, or even panics. Where error messages exist, they may be
+misleading. Finally, the languages maintainers may be less familiar with how to
+solve the problems, and/or rely on less available key maintainers.
+
+### Mitigating Constraints
+
+The above constraints affect the library design and dependency choices in your
+source, and by extension the choices of library dependencies you can use. In
+extreme cases, constraints or support concerns may lead developers to choose
+newer languages like [Zig][10].
+
+Regardless of the programming language used, the best advice is to unit test
+your code, and run tests with your intended WebAssembly runtime, like wazero.
+
+These tests should cover the critical paths of your code, including errors.
+Doing so protects your time. You'll have higher confidence, and more efficient
+means to communicate problems vs ad-hoc reports.
+
+## System Calls
+
+WebAssembly is a stack-based virtual machine specification, so operates at a
+lower level than an operating system. For functionality the operating system
+would otherwise provide, system interfaces are needed.
+
+Programming languages usually include a standard library, with features that
+require I/O, such as writing to the console. Portability is helped along with
+[POSIX][3] conforming implementations of system calls, such as `fd_read`.
+
+There is a [WebAssembly System Interface]({{< ref "/specs#wasi" >}}), a.k.a.
+WASI, which defines host functions loosely based on POSIX. There's also a
+de facto implementation [wasi-libc][4]. However, WASI is not a standard and
+language compilers don't always support it.
+
+For example, AssemblyScript once supported WASI, but no longer does. Even
+compilers that target WASI using [wasi-libc][4] have gaps. For example,
+[TinyGo](tinygo) does not yet support `fd_readdir`. Some toolchains have a
+hybrid approach. For example, Emscripten uses WASI for console output, but its
+own virtual filesystem functions. Finally, the team behind WASI are
+developing an incompatible, modular replacement to the current version.
+
+It is important to note that even when system interfaces are supported, some
+users prefer a freestanding compilation target that restricts them. This helps
+them control binary size and performance.
+
+In summary, system interfaces in WebAssembly are not standard and are immature.
+Developers need to understand and test the system interfaces they rely on.
+Testing ensures not only the present capabilities, but also they continue to
+operate as the ecosystem matures.
+
 ## Concurrency
 
 WebAssembly does not yet support true parallelism; it lacks support for
 multiple threads, atomics, and memory barriers. (It may someday; See
 the [threads proposal][5].)
 
-For example, a compiler targeting [WASI][3], generates a `_start` function
-corresponding to `main` in the original source code. When the WebAssembly
-runtime calls `_start`, it remains on the same thread of execution until that
-function completes.
+For example, a compiler targeting [WASI]({{< ref "/specs#wasi" >}}), generates
+a `_start` function corresponding to `main` in the original source code. When
+the WebAssembly runtime calls `_start`, it remains on the same thread of
+execution until that function completes.
 
 Concretely, if using wazero, a Wasm function call remains on the calling
 goroutine until it completes.
 
 In summary, while true that host functions can do anything, including launch
-processes, Wasm binaries compliant with [WebAssembly Core 2.0][4] cannot do
-anything in parallel, unless they use non-standard instructions or conventions
-not yet defined by the specification.
+processes, Wasm binaries compliant with the [WebAssembly Core Specification]
+({{< ref "/specs#core" >}}) cannot do anything in parallel, unless they use
+non-standard instructions or conventions not yet defined by the specification.
 
 ### Compiling Parallel Code to Serial Wasm
 
@@ -78,10 +145,11 @@ invoked in parallel, despite not being able to share memory.
 
 [1]: https://github.com/tetratelabs/wazero/tree/main/site/content/languages
 [2]: https://github.com/tetratelabs/wazero/stargazers
-[3]: https://github.com/WebAssembly/WASI/blob/snapshot-01/design/application-abi.md#current-unstable-abi
-[4]: https://www.w3.org/TR/2022/WD-wasm-core-2-20220419/
+[3]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/contents.html
+[4]: https://github.com/WebAssembly/wasi-libc
 [5]: https://github.com/WebAssembly/threads
 [6]: https://llvm.org
 [7]: https://github.com/WebAssembly/binaryen
 [8]: https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp
 [9]: https://github.com/wapc/wapc-go
+[10]: https://ziglang.org/
diff --git a/site/content/languages/go.md b/site/content/languages/go.md
index 92663a8e..a10688a1 100644
--- a/site/content/languages/go.md
+++ b/site/content/languages/go.md
@@ -36,6 +36,11 @@ features.
 
 ## Constraints
 
+Please read our overview of WebAssembly and
+[constraints]({{< ref "_index.md#constraints" >}}). In short, expect
+limitations in both language features and library choices when developing your
+software.
+
 `GOARCH=wasm GOOS=js` has a custom ABI which supports a subset of features in
 the Go standard library. Notably, the host can implement time, crypto, file
 system and HTTP client functions. Even where implemented, certain operations
@@ -43,15 +48,8 @@ will have no effect for reasons like ignoring HTTP request properties or fake
 values returned (such as the pid). When not supported, many functions return
 `syscall.ENOSYS` errors, or the string form: "not implemented on js".
 
-The impact of this is that end users should expect that compilation is not
-enough to ensure a program works in wasm. Use of fake properties or
-unimplemented API usage can result in invalid behavior or errors at runtime.
-For this reason, all code compiled to wasm should be unit tested with the same
-runtime configuration expected in production.
-
 Here are the more notable parts of Go which will not work when compiled via
 `GOARCH=wasm GOOS=js`, resulting in `syscall.ENOSYS` errors:
-* Goroutines. Ex. `go func(){}()`
 * Raw network access. Ex. `net.Bind`
 * File descriptor control (`fnctl`). Ex. `syscall.Pipe`
 * Arbitrary syscalls. Ex `syscall.Syscall`
@@ -68,9 +66,14 @@ begins at [ld.wasmMinDataAddr][12], offset 12288.
 
 ## System Calls
 
-"syscall/js.*" are host functions for managing the JavaScript object graph,
-including functions to make and finalize objects, arrays and numbers
-(`js.Value`).
+Please read our overview of WebAssembly and
+[System Calls]({{< ref "_index.md#system-calls" >}}). In short, WebAssembly is
+a stack-based virtual machine specification, so operates at a lower level than
+an operating system.
+
+"syscall/js.*" are host functions for features the operating system would
+otherwise provide. These also manage the JavaScript object graph, including
+functions to make and finalize objects, arrays and numbers (`js.Value`).
 
 Each `js.Value` has a `js.ref`, which is either a numeric literal or an object
 reference depending on its 64-bit bit pattern. When an object, the first 31
diff --git a/site/content/languages/rust.md b/site/content/languages/rust.md
index ed651833..79d779a6 100644
--- a/site/content/languages/rust.md
+++ b/site/content/languages/rust.md
@@ -21,8 +21,9 @@ brevity.
 ## Overview
 
 When Rust compiles a `%.rs` file with a `wasm32-*` target, the output `%.wasm`
-depends on a subset of features in the [WebAssembly 1.0 Core specification][2].
-The `wasm32-wasi` target depends on [WASI][3] host functions as well.
+depends on a subset of features in the [WebAssembly 1.0 Core specification]
+({{< ref "/specs#core" >}}). The `wasm32-wasi` target depends on [WASI]
+({{< ref "/specs#wasi" >}}) host functions as well.
 
 Unlike some compilers, Rust also supports importing custom host functions and
 exporting functions back to the host.
@@ -67,9 +68,10 @@ the next person.
 
 ## Constraints
 
-Like other compilers that can target wasm, there are constraints using Rust.
-These constraints affect the library design and dependency choices in your
-source.
+Please read our overview of WebAssembly and
+[constraints]({{< ref "_index.md#constraints" >}}). In short, expect
+limitations in both language features and library choices when developing your
+software.
 
 The most common constraint is which crates you can depend on. Please refer to
 the [Which Crates Will Work Off-the-Shelf with WebAssembly?][8] page in the
@@ -148,10 +150,14 @@ from your business logic as much as possible.
 
 ## System Calls
 
-WebAssembly is a stack-based virtual machine specification, so operates at a
-lower level than an operating system. For functionality the operating system
-would otherwise provide, you must use the `wasm32-wasi` target. This imports
-host functions defined in [WASI][3], described in [Specifications]({{< ref "/specs" >}}).
+Please read our overview of WebAssembly and
+[System Calls]({{< ref "_index.md#system-calls" >}}). In short, WebAssembly is
+a stack-based virtual machine specification, so operates at a lower level than
+an operating system.
+
+For functionality the operating system would otherwise provide, you must use
+the `wasm32-wasi` target. This imports host functions in
+[WASI]({{< ref "/specs#wasi" >}}).
 
 For example, `rustc -o hello.wasm --target wasm32-wasi hello.rs` compiles the
 below `main` function into a WASI function exported as `_start`.
@@ -210,8 +216,6 @@ performance for a [smaller binary](#binary-size). After that, tuning your
 source code may reduce binary size further.
 
 [1]: https://www.rust-lang.org/tools/install
-[2]: https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/
-[3]: https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md
 [4]: https://docs.rust-embedded.org/book/interoperability/rust-with-c.html#no_mangle
 [5]: https://rustwasm.github.io/docs/book
 [6]: https://github.com/tetratelabs/wazero/tree/main/site/content/languages/rust.md
diff --git a/site/content/languages/tinygo.md b/site/content/languages/tinygo.md
index ab499a59..458df810 100644
--- a/site/content/languages/tinygo.md
+++ b/site/content/languages/tinygo.md
@@ -17,8 +17,9 @@ embeds in Go applications. Hence, all notes below will be about TinyGo's
 ## Overview
 
 When TinyGo compiles a `%.go` file with its `wasi` target, the output `%.wasm`
-depends on a subset of features in the [WebAssembly 2.0 Core specification][2],
-as well [WASI][3] host functions.
+depends on a subset of features in the [WebAssembly 2.0 Core specification]
+({{< ref "/specs#core" >}}) and [WASI]({{< ref "/specs#wasi" >}}) host
+functions.
 
 Unlike some compilers, TinyGo also supports importing custom host functions and
 exporting functions back to the host.
@@ -55,11 +56,15 @@ the next person.
 
 ## Constraints
 
-Like other compilers that can target wasm, there are constraints using TinyGo.
-These constraints affect the library design and dependency choices in your Go
-source.
+Please read our overview of WebAssembly and
+[constraints]({{< ref "_index.md#constraints" >}}). In short, expect
+limitations in both language features and library choices when developing your
+software.
+
+### Unsupported standard libraries
 
-### Partial Reflection Support
+TinyGo does not completely implement the Go standard library when targeting
+`wasi`. What is missing is documented [here][26].
 
 The first constraint people notice is that `encoding/json` usage compiles, but
 panics at runtime.
@@ -83,7 +88,7 @@ This is due to limited support for reflection, and effects other [serialization
 tools][18] also. See [Frequently Asked Questions](#frequently-asked-questions)
 for some workarounds.
 
-### Unimplemented System Calls
+### Unsupported System Calls
 
 You may also notice some other features not yet work. For example, the below
 will compile, but print "readdir unimplemented : errno 54" at runtime.
@@ -104,17 +109,6 @@ The underlying error is often, but not always `syscall.ENOSYS` which is the
 standard way to stub a syscall until it is implemented. If you are interested
 in more, see [System Calls](#system-calls).
 
-### Mitigating Constraints
-
-Realities like this are not unique to TinyGo as they will happen compiling any
-language not written specifically with WebAssembly in mind. Knowing the same
-code compiled to wasm may return errors or worse panic, the main mitigation
-approach is testing.
-
-Unit test the critical paths of your code, including errors, on your target
-WebAssembly runtime, such as wazero. This not only gives higher confidence, but
-is also a much more efficient means to communicate bugs vs ad-hoc reports.
-
 ## Memory
 
 When TinyGo compiles go into wasm, it configures the WebAssembly linear memory
@@ -222,10 +216,13 @@ approach later.
 
 ## System Calls
 
-WebAssembly is a stack-based virtual machine specification, so operates at a
-lower level than an operating system. For functionality the operating system
-would otherwise provide, TinyGo imports host functions defined in [WASI][3],
-described in [Specifications]({{< ref "/specs" >}}).
+Please read our overview of WebAssembly and
+[System Calls]({{< ref "_index.md#system-calls" >}}). In short, WebAssembly is
+a stack-based virtual machine specification, so operates at a lower level than
+an operating system.
+
+For functionality the operating system would otherwise provide, TinyGo imports
+host functions defined in [WASI]({{< ref "/specs#wasi" >}}).
 
 For example, `tinygo build -o main.wasm -target=wasi main.go` compiles the
 below `main` function into a WASI function exported as `_start`.
@@ -343,7 +340,8 @@ Meanwhile, most users resort to non-reflective parsers, such as [gjson][17].
 
 ### Why does my wasm import WASI functions even when I don't use it?
 TinyGo has a `wasm` target (for browsers) and a `wasi` target for runtimes that
-support [WASI][3]. This document is written only about the `wasi` target.
+support [WASI]({{< ref "/specs#wasi" >}}). This document is written only about
+the `wasi` target.
 
 Some users are surprised to see imports from WASI (`wasi_snapshot_preview1`),
 when their neither has a main function nor uses memory. At least implementing
@@ -363,8 +361,6 @@ surprising to users are APIs that seem simple, but require a lot of supporting
 functions, such as `fmt.Println`, which can require 100KB of wasm.
 
 [1]: https://tinygo.org/
-[2]: https://www.w3.org/TR/2022/WD-wasm-core-2-20220419/
-[3]: https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md
 [4]: https://tinygo.org/docs/guides/webassembly/
 [5]: https://github.com/tinygo-org/tinygo#getting-help
 [6]: https://github.com/tetratelabs/wazero/tree/main/site/content/languages/tinygo.md
@@ -387,3 +383,4 @@ functions, such as `fmt.Println`, which can require 100KB of wasm.
 [23]: https://github.com/WebAssembly/binaryen/blob/main/src/passes/Asyncify.cpp
 [24]: http://tleyden.github.io/blog/2014/10/30/goroutines-vs-threads/
 [25]: https://github.com/tinygo-org/tinygo/issues/3095
+[26]: https://tinygo.org/docs/reference/lang-support/stdlib/
diff --git a/site/content/specs.md b/site/content/specs.md
index c0c5d047..4cba286b 100644
--- a/site/content/specs.md
+++ b/site/content/specs.md
@@ -31,7 +31,7 @@ such as AssemblyScript.
 In summary, we hope this section can guide you in terms of what wazero supports
 as well as how to classify a request for a feature we don't yet support.
 
-### WebAssembly Core
+### WebAssembly Core {#core}
 wazero conforms with tests defined alongside WebAssembly Core
 Specification [1.0][1] and [2.0][14].
 
@@ -56,7 +56,7 @@ Features not yet assigned to a W3C release are not reliable. Encourage the
 [WebAssembly community][12] to formalize features you rely on, so that they
 become assigned to a release, and reach the W3C recommendation (REC) phase.
 
-### WebAssembly System Interface (WASI)
+### WebAssembly System Interface (WASI) {#wasi}
 
 Many compilers implement system calls using the WebAssembly System Interface,
 [WASI][5]. WASI is a WebAssembly community [subgroup][3], but has not published


Test patch:
diff --git a/internal/compilationcache/file_cache_test.go b/internal/compilationcache/file_cache_test.go
index 84e508ea..c9cdd4bf 100644
--- a/internal/compilationcache/file_cache_test.go
+++ b/internal/compilationcache/file_cache_test.go
@@ -2,8 +2,11 @@ package compilationcache
 
 import (
 	"bytes"
+	"fmt"
 	"io"
 	"os"
+	"runtime"
+	"strings"
 	"testing"
 
 	"github.com/tetratelabs/wazero/internal/testing/require"
@@ -133,3 +136,11 @@ func TestFileCache_Get(t *testing.T) {
 		require.False(t, ok)
 	})
 }
+
+func TestFileCache_path(t *testing.T) {
+	fc := &fileCache{dirPath: "/tmp/.wazero"}
+	actual := fc.path(Key{1, 2, 3, 4, 5})
+	require.Contains(t, actual, fmt.Sprintf("%s-%s-", runtime.GOARCH, runtime.GOOS))
+	require.True(t, strings.HasPrefix(actual, fc.dirPath))
+	require.True(t, strings.HasSuffix(actual, "0102030405000000000000000000000000000000000000000000000000000000"))
+}