Repository: vega/vl-convert (language: python)

Issue:
deno_fetch BorrowMutError with 0.11.1
After updating from 0.10.3 to 0.11.1, the Altair CI tests are failing flakily with this error:

```
thread 'tokio-runtime-worker' panicked at 'already borrowed: BorrowMutError', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_fetch-0.132.0/lib.rs:453:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(262), ...)', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.190.0/runtime.rs:2432:50

stack backtrace:
   0:        0x178465454 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x178071b18 - _BrotliDecoderVersion
   2:        0x1784438c4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x178469384 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x178468fe4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x178469ed0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x178469a80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x1784699f0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x1784699e4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x17958ca1c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x17958cc68 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x17810410c - _BrotliDecoderVersion
  12:        0x1781068d0 - _BrotliDecoderVersion
  13:        0x17809f8f8 - _BrotliDecoderVersion
  14:        0x1780b6a70 - _BrotliDecoderVersion
  15:        0x178493b54 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  16:        0x178492aa8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x1784a05a0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  18:        0x17849801c - _rustsecp256k1_v0_8_1_default_error_callback_fn
  19:        0x178497bf0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  20:        0x178498580 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  21:        0x17846bc80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  22:        0x19c227fa8 - __pthread_joiner_wake
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(137), ...)', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.190.0/runtime.rs:2432:50
stack backtrace:
   0:        0x178465454 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x178071b18 - _BrotliDecoderVersion
   2:        0x1784438c4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x178469384 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x178468fe4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x178469ed0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x178469a80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x1784699f0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x1784699e4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x17958ca1c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x17958cc68 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x1780a923c - _BrotliDecoderVersion
  12:        0x17851b9cc - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  13:        0x17851d06c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  14:        0x178516f50 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  15:        0x178529b08 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  16:        0x17846bc80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x19c227fa8 - __pthread_joiner_wake
```

This is the error both locally on an M1 mac and on CI on linux-64
deno_fetch BorrowMutError with 0.11.1
After updating from 0.10.3 to 0.11.1, the Altair CI tests are failing flakily with this error:

```
thread 'tokio-runtime-worker' panicked at 'already borrowed: BorrowMutError', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_fetch-0.132.0/lib.rs:453:6
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(262), ...)', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.190.0/runtime.rs:2432:50

stack backtrace:
   0:        0x178465454 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x178071b18 - _BrotliDecoderVersion
   2:        0x1784438c4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x178469384 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x178468fe4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x178469ed0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x178469a80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x1784699f0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x1784699e4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x17958ca1c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x17958cc68 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x17810410c - _BrotliDecoderVersion
  12:        0x1781068d0 - _BrotliDecoderVersion
  13:        0x17809f8f8 - _BrotliDecoderVersion
  14:        0x1780b6a70 - _BrotliDecoderVersion
  15:        0x178493b54 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  16:        0x178492aa8 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x1784a05a0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  18:        0x17849801c - _rustsecp256k1_v0_8_1_default_error_callback_fn
  19:        0x178497bf0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  20:        0x178498580 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  21:        0x17846bc80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  22:        0x19c227fa8 - __pthread_joiner_wake
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(137), ...)', /Users/jonmmease/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deno_core-0.190.0/runtime.rs:2432:50
stack backtrace:
   0:        0x178465454 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   1:        0x178071b18 - _BrotliDecoderVersion
   2:        0x1784438c4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   3:        0x178469384 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   4:        0x178468fe4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   5:        0x178469ed0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   6:        0x178469a80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   7:        0x1784699f0 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   8:        0x1784699e4 - _rustsecp256k1_v0_8_1_default_error_callback_fn
   9:        0x17958ca1c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  10:        0x17958cc68 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  11:        0x1780a923c - _BrotliDecoderVersion
  12:        0x17851b9cc - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  13:        0x17851d06c - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  14:        0x178516f50 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  15:        0x178529b08 - _v8_inspector__V8InspectorClient__BASE__consoleAPIMessage
  16:        0x17846bc80 - _rustsecp256k1_v0_8_1_default_error_callback_fn
  17:        0x19c227fa8 - __pthread_joiner_wake
```

This is the error both locally on an M1 mac and on CI on linux-64


Discussion / hints:
Reported in https://github.com/denoland/deno/issues/19670
Reported in https://github.com/denoland/deno/issues/19670

Fix patch:
diff --git a/Cargo.lock b/Cargo.lock
index 75874d5a..0bea2e9c 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -5700,6 +5700,7 @@ dependencies = [
  "lazy_static",
  "pyo3",
  "pythonize",
+ "tokio",
  "vl-convert-rs",
 ]
 
diff --git a/vl-convert-python/Cargo.toml b/vl-convert-python/Cargo.toml
index 43004a5b..05dc4559 100644
--- a/vl-convert-python/Cargo.toml
+++ b/vl-convert-python/Cargo.toml
@@ -22,4 +22,5 @@ pyo3 = { version = "0.19.0", features = ["extension-module", "anyhow", "abi3-py3
 lazy_static = "1.4.0"
 futures = "0.3.24"
 pythonize = "0.19.0"
+tokio = {version= "1.21", features=["rt-multi-thread"]}
 
diff --git a/vl-convert-python/src/lib.rs b/vl-convert-python/src/lib.rs
index 19a77cd0..76109710 100644
--- a/vl-convert-python/src/lib.rs
+++ b/vl-convert-python/src/lib.rs
@@ -4,7 +4,7 @@ use pyo3::types::{PyBytes, PyDict};
 use pythonize::{depythonize, pythonize};
 use std::str::FromStr;
 use std::sync::Mutex;
-use vl_convert_rs::converter::{VlOpts, TOKIO_RUNTIME};
+use vl_convert_rs::converter::VlOpts;
 use vl_convert_rs::module_loader::import_map::VlVersion;
 use vl_convert_rs::serde_json;
 use vl_convert_rs::text::register_font_directory as register_font_directory_rs;
@@ -15,6 +15,11 @@ extern crate lazy_static;
 
 lazy_static! {
     static ref VL_CONVERTER: Mutex<VlConverterRs> = Mutex::new(VlConverterRs::new());
+    static ref PYTHON_RUNTIME: tokio::runtime::Runtime =
+        tokio::runtime::Builder::new_multi_thread()
+            .enable_all()
+            .build()
+            .unwrap();
 }
 
 /// Convert a Vega-Lite spec to a Vega spec using a particular
@@ -48,7 +53,7 @@ fn vegalite_to_vega(
     let mut converter = VL_CONVERTER
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
-    let vega_spec = match TOKIO_RUNTIME.block_on(converter.vegalite_to_vega(
+    let vega_spec = match PYTHON_RUNTIME.block_on(converter.vegalite_to_vega(
         vl_spec,
         VlOpts {
             vl_version,
@@ -85,7 +90,7 @@ fn vega_to_svg(vg_spec: PyObject) -> PyResult<String> {
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
 
-    let svg = match TOKIO_RUNTIME.block_on(converter.vega_to_svg(vg_spec)) {
+    let svg = match PYTHON_RUNTIME.block_on(converter.vega_to_svg(vg_spec)) {
         Ok(vega_spec) => vega_spec,
         Err(err) => {
             return Err(PyValueError::new_err(format!(
@@ -129,7 +134,7 @@ fn vegalite_to_svg(
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
 
-    let svg = match TOKIO_RUNTIME.block_on(converter.vegalite_to_svg(
+    let svg = match PYTHON_RUNTIME.block_on(converter.vegalite_to_svg(
         vl_spec,
         VlOpts {
             vl_version,
@@ -165,7 +170,7 @@ fn vega_to_png(vg_spec: PyObject, scale: Option<f32>) -> PyResult<PyObject> {
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
 
-    let png_data = match TOKIO_RUNTIME.block_on(converter.vega_to_png(vg_spec, scale)) {
+    let png_data = match PYTHON_RUNTIME.block_on(converter.vega_to_png(vg_spec, scale)) {
         Ok(vega_spec) => vega_spec,
         Err(err) => {
             return Err(PyValueError::new_err(format!(
@@ -214,7 +219,7 @@ fn vegalite_to_png(
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
 
-    let png_data = match TOKIO_RUNTIME.block_on(converter.vegalite_to_png(
+    let png_data = match PYTHON_RUNTIME.block_on(converter.vegalite_to_png(
         vl_spec,
         VlOpts {
             vl_version,
@@ -289,7 +294,7 @@ fn get_local_tz() -> PyResult<Option<String>> {
     let mut converter = VL_CONVERTER
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
-    let local_tz = match TOKIO_RUNTIME.block_on(converter.get_local_tz()) {
+    let local_tz = match PYTHON_RUNTIME.block_on(converter.get_local_tz()) {
         Ok(local_tz) => local_tz,
         Err(err) => {
             return Err(PyValueError::new_err(format!(
@@ -311,7 +316,7 @@ fn get_themes() -> PyResult<PyObject> {
     let mut converter = VL_CONVERTER
         .lock()
         .expect("Failed to acquire lock on Vega-Lite converter");
-    let themes = match TOKIO_RUNTIME.block_on(converter.get_themes()) {
+    let themes = match PYTHON_RUNTIME.block_on(converter.get_themes()) {
         Ok(themes) => themes,
         Err(err) => {
             return Err(PyValueError::new_err(format!(
diff --git a/vl-convert-rs/src/converter.rs b/vl-convert-rs/src/converter.rs
index 537b255b..9b07ba5f 100644
--- a/vl-convert-rs/src/converter.rs
+++ b/vl-convert-rs/src/converter.rs
@@ -31,7 +31,7 @@ use crate::text::{op_text_width, FONT_DB, USVG_OPTIONS};
 
 lazy_static! {
     pub static ref TOKIO_RUNTIME: tokio::runtime::Runtime =
-        tokio::runtime::Builder::new_multi_thread()
+        tokio::runtime::Builder::new_current_thread()
             .enable_all()
             .build()
             .unwrap();
@@ -553,42 +553,43 @@ impl VlConverter {
         let (sender, mut receiver) = mpsc::channel::<VlConvertCommand>(32);
 
         let handle = Arc::new(thread::spawn(move || {
-            let mut inner = TOKIO_RUNTIME.block_on(InnerVlConverter::try_new())?;
-
-            while let Some(cmd) = TOKIO_RUNTIME.block_on(receiver.next()) {
-                match cmd {
-                    VlConvertCommand::VlToVg {
-                        vl_spec,
-                        vl_opts,
-                        responder,
-                    } => {
-                        let vega_spec =
-                            TOKIO_RUNTIME.block_on(inner.vegalite_to_vega(&vl_spec, vl_opts));
-                        responder.send(vega_spec).ok();
-                    }
-                    VlConvertCommand::VgToSvg { vg_spec, responder } => {
-                        let svg_result = TOKIO_RUNTIME.block_on(inner.vega_to_svg(&vg_spec));
-                        responder.send(svg_result).ok();
-                    }
-                    VlConvertCommand::VlToSvg {
-                        vl_spec,
-                        vl_opts,
-                        responder,
-                    } => {
-                        let svg_result =
-                            TOKIO_RUNTIME.block_on(inner.vegalite_to_svg(&vl_spec, vl_opts));
-                        responder.send(svg_result).ok();
-                    }
-                    VlConvertCommand::GetLocalTz { responder } => {
-                        let local_tz = TOKIO_RUNTIME.block_on(inner.get_local_tz());
-                        responder.send(local_tz).ok();
-                    }
-                    VlConvertCommand::GetThemes { responder } => {
-                        let themes = TOKIO_RUNTIME.block_on(inner.get_themes());
-                        responder.send(themes).ok();
+            TOKIO_RUNTIME.block_on(async {
+                let mut inner = InnerVlConverter::try_new().await?;
+                while let Some(cmd) = receiver.next().await {
+                    match cmd {
+                        VlConvertCommand::VlToVg {
+                            vl_spec,
+                            vl_opts,
+                            responder,
+                        } => {
+                            let vega_spec = inner.vegalite_to_vega(&vl_spec, vl_opts).await;
+                            responder.send(vega_spec).ok();
+                        }
+                        VlConvertCommand::VgToSvg { vg_spec, responder } => {
+                            let svg_result = inner.vega_to_svg(&vg_spec).await;
+                            responder.send(svg_result).ok();
+                        }
+                        VlConvertCommand::VlToSvg {
+                            vl_spec,
+                            vl_opts,
+                            responder,
+                        } => {
+                            let svg_result = inner.vegalite_to_svg(&vl_spec, vl_opts).await;
+                            responder.send(svg_result).ok();
+                        }
+                        VlConvertCommand::GetLocalTz { responder } => {
+                            let local_tz = inner.get_local_tz().await;
+                            responder.send(local_tz).ok();
+                        }
+                        VlConvertCommand::GetThemes { responder } => {
+                            let themes = inner.get_themes().await;
+                            responder.send(themes).ok();
+                        }
                     }
                 }
-            }
+                Ok::<(), AnyError>(())
+            })?;
+
             Ok(())
         }));
 


Test patch:
diff --git a/vl-convert-python/tests/test_specs.py b/vl-convert-python/tests/test_specs.py
index c1814ca3..496ca44b 100644
--- a/vl-convert-python/tests/test_specs.py
+++ b/vl-convert-python/tests/test_specs.py
@@ -85,11 +85,11 @@ def test_svg(name, as_dict):
     # Convert to vega first
     vg_spec = vlc.vegalite_to_vega(vl_spec, vl_version=vl_version)
     svg = vlc.vega_to_svg(vg_spec)
-    assert svg == expected_svg
+    check_svg(svg, expected_svg)
 
     # Convert directly to image
     svg = vlc.vegalite_to_svg(vl_spec, vl_version=vl_version)
-    assert svg == expected_svg
+    check_svg(svg, expected_svg)
 
 
 @pytest.mark.parametrize(
@@ -100,6 +100,7 @@ def test_svg(name, as_dict):
         ("remote_images", 1.0),
         ("maptile_background", 1.0),
         ("no_text_in_font_metrics", 1.0),
+        ("lookup_urls", 1.0),
     ],
 )
 @pytest.mark.parametrize("as_dict", [False])
@@ -143,9 +144,27 @@ def test_png_theme_config(name, scale, theme):
     assert png == expected_png
 
 
+def test_gh_78():
+    vl_version = "v5_8"
+    name = "lookup_urls"
+    vl_spec = json.loads(load_vl_spec(name))
+
+    png = None
+    for i in range(30):
+        png = vlc.vegalite_to_png(vl_spec, vl_version=vl_version)
+
+    expected_png = load_expected_png(name, vl_version)
+    check_png(png, expected_png)
+
+
 def check_png(png, expected_png):
     png_img = imread(BytesIO(png))
     expected_png_img = imread(BytesIO(expected_png))
     similarity_value = ssim(png_img, expected_png_img, channel_axis=2)
     if similarity_value < 0.995:
         pytest.fail(f"png mismatch with similarity: {similarity_value}")
+
+
+def check_svg(svg, expected_svg):
+    if svg != expected_svg:
+        pytest.fail(f"svg image mismatch")
diff --git a/vl-convert-rs/tests/vl-specs/expected/v5_8/lookup_urls.png b/vl-convert-rs/tests/vl-specs/expected/v5_8/lookup_urls.png
new file mode 100644
index 00000000..b2a5763b
Binary files /dev/null and b/vl-convert-rs/tests/vl-specs/expected/v5_8/lookup_urls.png differ
diff --git a/vl-convert-rs/tests/vl-specs/lookup_urls.vl.json b/vl-convert-rs/tests/vl-specs/lookup_urls.vl.json
new file mode 100644
index 00000000..adc9b593
--- /dev/null
+++ b/vl-convert-rs/tests/vl-specs/lookup_urls.vl.json
@@ -0,0 +1,32 @@
+{
+  "config": {"view": {"continuousWidth": 300, "continuousHeight": 300}},
+  "data": {
+    "url": "https://cdn.jsdelivr.net/npm/vega-datasets@v1.29.0/data/income.json"
+  },
+  "mark": {"type": "geoshape"},
+  "encoding": {
+    "color": {"field": "pct", "type": "quantitative"},
+    "shape": {"field": "geo", "type": "geojson"},
+    "tooltip": [
+      {"field": "name", "type": "nominal"},
+      {"field": "pct", "type": "quantitative"}
+    ]
+  },
+  "height": 175,
+  "projection": {"type": "albersUsa"},
+  "transform": [
+    {
+      "lookup": "id",
+      "as": "geo",
+      "from": {
+        "data": {
+          "url": "https://cdn.jsdelivr.net/npm/vega-datasets@v1.29.0/data/us-10m.json",
+          "format": {"feature": "states", "type": "topojson"}
+        },
+        "key": "id"
+      }
+    }
+  ],
+  "width": 300,
+  "$schema": "https://vega.github.io/schema/vega-lite/v5.8.0.json"
+}