Issue 1:
I can't postmessage a shared WebAssembly memory or module to a worker:

results in:

Same happens with wasm module, also I cant use any of them to move wasm execution into a worker

Do I need a special flag for this?

Command I used to run this:

docker run -it --init -v ${PWD}:/app hayd/alpine-deno run --allow-read --unstable --v8-flags="--experimental-wasm-threads" /app/main.ts

test.zip

Issue 2:
Deno op dispatch and native plugins

Overview

Deno dispatch system is based on

Deno.core.send(opId: number, control: null | ArrayBufferView, data?: ArrayBufferView)

a function that passes messages from JS to Rust and executes function described with opId.

Message passing is done using so-called "shared queue", which for now has served it's purpose well.

The opId which identifies the op and its dispatcher is a number that is shared between TypeScript and Rust.

As of Deno v0.18.0 all ops dispatching logic is hardcoded.

Problem

Current solution is very simple and quite efficient, however has several shortcomings that need to be addressed

to expose native plugins API to users;

OpId has to be manually synchronized between TS and Rust. In current situation, where we have about 60 ops, it's not that bad, but surely we'll be cumbersome as number of ops grows.

...
// Warning! These values are duplicated in the TypeScript code (js/dispatch.ts),
// update with care.
pub const OP_READ: OpId = 1;
pub const OP_WRITE: OpId = 2;
...

There is no way to add new op programatically. Op has be to registered in code manually before compilation, which prevents users from adding new ops from their' code.

let op = match op_id {
OP_READ => {
dispatch_minimal::dispatch(io::op_read, state, control, zero_copy)
}
OP_WRITE => {
dispatch_minimal::dispatch(io::op_write, state, control, zero_copy)
}
...
}

Op and its dispatcher are very tightly coupled. There still be tight coupling between op and it's dispatcher (you can't easily write op that will work with both "minimal" and "json" dispatcher), but it should be done using a simple API instead of manually writing closures.

// see example above

Code for ops lives in 2 different places. TS code for ops currently lives in //js/ while Rust code lives in //cli/ops/. Ideally we'd like to have //ops directory that will contain both TS and Rust code (eg. //ops/fs/main.ts and //ops/fs/mod.ts as entry points).

Nomenclature

dispatch - main method used to call into Rust code from JS. See Isolate::set_dispatch method.

dispatcher - an abstract construct that is called from dispatch. It's aware of serialization method ("minimal", "json") and sets contract for op handler.

op/op handler - a function that performs some actual work, eg. "read"/"write"/"dial". Op has to be used with a dispatcher that knows how to transform result of that op into CoreOp.

CoreOp - fundamental type that describes how Rust code can return value to JS.

Requirements

Minimal or no changes to Deno.core and shared queue infrastructure.

New system must be FAST (at least as fast as current solution, preferably faster).

Ops should have access to permission APIs (TODO: can permissions be forced upon plugins/ops?)

Op handler signature shouldn't depend on ThreadSafeState because it's a deno_cli concept.

Ability to register new dispatcher with custom serialization method (eg. "msgpack").

Ability to register new op statically (before build).

No hardcoded opIds - after registration of an op, during startup Rust sends an "op map" that is a dictonary mapping op names to op ids.

Ability to register new op dynamically (from JS code), loading some Rust code (as dynamic library?). This functionality can be exposed as another op - op_register_new_op.

Ability to register an op multiple times. Say I have some native plugin that requires "read" op, but I'm not sure that host will have "read" available. So I register the op in my library setup code (potentially with namespace of my plugin).

Both static and dynamic op registration should be as simple as possible.

Proposed dispatch implementation

Prototype by @afinch7

In #2840 there's a complete solution.

"op listeners"

After registration of a new op a bunch of listener methods is called to reset the ids of all ops.

I believe this can be refactored to encapsulate this logic in JsonOp (presented in "pros" section) and ultimately

should be handled by separate op that calls "callback" function on dispatch to register op ids that are sent from Rust.

This is only done when using set_dispatch_registry, which can be combined with set_dispatch. It's functionality is equivalent

to proposed setOpMap below.

dispatcher registry

pub trait OpDispatcher: Send + Sync {
fn dispatch(&self, args: &[u8], buf: Option<PinnedBuf>) -> CoreOp;
}

type OpDispatcherRegistry = Vec<Option<Arc<Box<dyn OpDispatcher>>>>;

pub struct OpDisReg {
// Quick lookups by unique "op id"/"resource id"
// The main goal of op_dis_registry is to perform lookups as fast
// as possible at all times.
op_dis_registry: RwLock<OpDispatcherRegistry>,
next_op_dis_id: AtomicU32,
// Serves as "phone book" for op_dis_registry
// This should only be referenced for initial lookups. It isn't
// possible to achieve the level of perfromance we want if we
// have to query this for every dispatch, but it may be needed
// to build caches for subseqent lookups.
op_dis_id_registry: RwLock<HashMap<String, HashMap<&'static str, OpId>>>,
notifier_reg: RwLock<NotifyerReg>,
}

notifier_reg is used to trigger "op listeners" described above.

Pros:

elegant API for registration of ops in TS

const OP_CHDIR = new JsonOp(opNamespace, "chdir");

export function chdir(directory: string): void {
OP_CHDIR.sendSync({ directory });
}

It might need a little tweaking as each op would create a closure, but that has to verified experimentally.

nice code organization

All code (both Rust and TS) related to fs lives inside //ops/fs crate.

Cons:

each op is a trait

This leads for a lot of boilerplate code, where we essentialy have an empty structure that must implement a trait.

pub struct OpChmod;

#[derive(Deserialize)]
#[serde(rename_all = "camelCase")]
struct ChmodArgs {
promise_id: Option<u64>,
path: String,
mode: u32,
}

impl FsOpDispatcher for OpChmod {
fn dispatch(
&self,
state: &TSFsOpsState,
control: &[u8],
buf: Option<PinnedBuf>,
) -> CoreOp {
wrap_json_op(
move |args, _zero_copy| {
let args: ChmodArgs = serde_json::from_value(args)?;
let (path, path_) = deno_fs::resolve_from_cwd(args.path.as_ref())?;

state.check_write(&path_)?;

let is_sync = args.promise_id.is_none();
blocking_json(is_sync, move || {
debug!("op_chmod {}", &path_);
// Still check file/dir exists on windows
let _metadata = fs::metadata(&path).map_err(FsOpError::from)?;
#[cfg(any(unix))]
{
let mut permissions = _metadata.permissions();
permissions.set_mode(args.mode);
fs::set_permissions(&path, permissions).map_err(FsOpError::from)?;
}
Ok(json!({}))
})
},
control,
buf,
)
}

const NAME: &'static str = "chmod";
}

a bit magical loading of TS code

A concept of "import map" (not import maps we have implemented in Deno tho!) is used to map bare specifier to path of .ts file inside the crate. Eg. "deno_op_fs" is mapped to "bundle/main.ts" in "ops_fs" crate.

Prototype by @bartlomieju

Taking a lot of inspiration from @afinch7 I came up with a bit different solution that allows to avoid problem of initial double lookup.

// NOTE: this can be merged into `ThreadSafeState` or `Isolate`
pub struct DispatchManager {
op_registry: RwLock<Vec<OpHandler>>
// TODO: HashMap signature should be changed to support namespacing
op_phone_book: RwLock<HashMap<String, (OpId, Dispatcher, Op)>>
}

In this case each registered op is placed in the op_phone_book structure to identify it by it's namespace and name and OpId is an offset in the op_registry vector obtained from length of the vector.

In the example above OpHandler type can be roughly described as:

// TODO: this has to be changed; we want to get rid of `state` and also add permission API
type OpHandler = Fn(state: &ThreadSafeState, control: &[u8], zero_copy: Option<PinnedBuf>) -> CoreOp);

Which is equivalent of wrapping any op with its designated dispatcher.

let read_handler = dispatch_minimal(op_read);
let resources_handler = dispatch_json(op_read);

// And registration...
isolate.register_op("read", read_handler);
isolate.register_op("resources", resources_handler);
// or
isolate.register_minimal_op("read", op_read);
isolate.register_json_op("resources", op_resources);

Then on startup a map of all registered ops can be sent to JS to synchronize JS counterparts functions:

// Construct an op map that looks like this:
// {
// "minimal": {
// "read": 1,
// "write": 2,
// },
// "json": {
// "resources": 3,
// "fs": {
// "chown": 4,
// "chmod": 5,
// ...
// }
// }
// }
let op_map = dispatch_manager.get_map()

isolate.execute(format!("denoMain({})", json!(op_map)));

function denoMain(opMao: OpMap, ...args): void {
dispatch.setOpMap(opMaps);
os.start(true, "TS");	os.start(true, "TS");
}

// dispatch.ts
export function setOpMap(opMap: OpMap): void {
// this can be further abstracted away
dispatch_minimal.setOpMap(opMap.minimal);
dispatch_json.setOpMap(opMap.json);
}

// dispatch_json.ts
class JsonOp {
namespace?: string;
name: string;
opId: number;

constructor(namespace: string, name?: string) {
if (!this.name) {
this.name = namespace;
} else {
this.namespace = namespace;
this.name = name;
}

registeredOp.add(this);
}

register(opId: number) {
this.opId = opId;
}

sendSync(...) {}

sendAsync(...) {}
}

const registeredOps = [
// store all instances of `JsonOp`
]

function setOpMap(opMap: opMap) {
for (const registeredOp in registeredOps) {
let opId;

let map = opMap;

if (registeredOp.namespace) {
map = opMap[registeredOp.namespace]
}

const opId = map[registeredOp.name];

if (!opId) {
throw new Error(`Unregistered op: ${registeredOp.name}`);
}

op.register(opId);
}
}

That way we get free validation that all ops are properly registered. Additional check can be added to ensure

that all ops sent from Rust are registered as well.

Upon dynamic registration of op we can do the same thing synchronously. The only constraint is that ops couldn't be unregistered

unless some synchronisation mechanism is added.

Questions/TODOs

Where should op be registered? @ry suggested doing it on Isolate which means ops become core concept (which in turn will require permissions to become core concept).

Proposed native plugins implementation

TODO

Starting this issue for further discussion.

This is at least partially 1.0 blocker (#2473)

Related issues:

#296

#2180

#1825

#2852

#2730

Related PRs:

#2385

#2785

#2811

#2840

#2939

CC @afinch7 @ry @piscisaureus @kevinkassimo