Repository: apache/arrow-rs (Rust)

Issue:
Variant: Read/Parse JSON value as Variant
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of https://github.com/apache/arrow-rs/issues/6736
- Depends on  https://github.com/apache/arrow-rs/issues/7424

A  major usecase for Variant values in Parquet and Arrow is efficiently
processing JSON encoded data. Thus an important capability is being able to
efficiently read JSON encoded bytes into the Variant binary encoding described in
[VariantEncoding.md]. This ticket covers an API to parse one JSON value to
one `Variant` value. Other tickets will cover converting converting Variants to
JSON as well as converting to/from Arrow Utf8* arrays and `Variant` arrays as
well as writing this to/from parquet.

[VariantEncoding.md]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md


**Describe the solution you'd like**

I would like an API to convert JSON encoded bytes to Variant encoded bytes



**Describe alternatives you've considered**


I suggest is an API like this:
```rust
// Provide location to write metadata, and value output
// (should be anything that implements `std::io::Write` or some trait)
let mut metadata_buffer = vec![];
let mut value_buffer = vec![];
// Input json encoded bytes
let json_data: &[u8] = ...;
// Call the new API
json_to_variant(&mut metadata_buffer, &mut value_buffer, json_data)?;
// metadata_buffer and value_buffer contain the variant information
```

Prior art:
- @ scovich's  PR here: #7403 
- 

**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->

# Considerations:

## Reusing metadata across values?

I think it will be common that same metadata is used across many different
variant values (e.g. because the schema of the json documents is the same). Thus
we should probably permit reusing validated metadata somehow (rather than
requiring recreating it for each decoded json value)

One option would be to add a json function to the `VariantBuilder` envisioned in
the ticket linked above for reading `Variant` values.

```rust
// Location to write metadata
let mut metadata_buffer = vec![]
// Create a builder
let builder = VariantBuilder::new(&mut metadata_buffer);
// Location to write the output variant value
let mut value_buffer = vec![];
builder.json(&mut value_buffer, json_data)?;
// value_buffer contains the result of converting json_data to `Variant`)
```

## Support for  "streaming"  / a push API

As sketched above, this API would require the entire JSON value in a single
buffer. A potentially more efficient API might be a "push" api, similar to how
the arrow JSON reader works, which would support smaller buffer sizes and lower
peak memory usage as well as interleaving variant parsing with IO fetch.

Perhaps something like

```rust
let mut metadata_buffer = vec![]
let builder = VariantBuilder::new(&mut metadata_buffer);
let mut value_buffer = vec![];
let mut parser = builder.json_parser(&mut value_buffer)?;
// json data comes in from some source
while let Some(json_data) = source.next() {
  parser.push(json_data); // incrementally parses json,
}
parser.finish(); // complete in-progress variant
// value_buffer contains the result of converting json_data to Variant)
```





Fix (patch):
diff --git a/parquet-variant/examples/variant_from_json_examples.rs b/parquet-variant/examples/variant_from_json_examples.rs
new file mode 100644
index 000000000000..e8a8a9d24959
--- /dev/null
+++ b/parquet-variant/examples/variant_from_json_examples.rs
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Example showing how to convert Variant values to JSON
+
+use parquet_variant::{
+    json_to_variant, variant_to_json, variant_to_json_string, variant_to_json_value, VariantBuilder,
+};
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let person_string = "{\"name\":\"Alice\", \"age\":30, ".to_string()
+        + "\"email\":\"alice@example.com\", \"is_active\": true, \"score\": 95.7,"
+        + "\"additional_info\": null}";
+
+    let mut variant_builder = VariantBuilder::new();
+    json_to_variant(&person_string, &mut variant_builder)?;
+
+    let (metadata, value) = variant_builder.finish();
+
+    let variant = parquet_variant::Variant::try_new(&metadata, &value)?;
+
+    let json_result = variant_to_json_string(&variant)?;
+    let json_value = variant_to_json_value(&variant)?;
+    let pretty_json = serde_json::to_string_pretty(&json_value)?;
+    println!("{pretty_json}");
+
+    let mut buffer = Vec::new();
+    variant_to_json(&mut buffer, &variant)?;
+    let buffer_result = String::from_utf8(buffer)?;
+    assert_eq!(json_result, "{\"additional_info\":null,\"age\":30,".to_string() +
+    "\"email\":\"alice@example.com\",\"is_active\":true,\"name\":\"Alice\",\"score\":95.7}");
+    assert_eq!(json_result, buffer_result);
+    assert_eq!(json_result, serde_json::to_string(&json_value)?);
+
+    Ok(())
+}
diff --git a/parquet-variant/src/builder.rs b/parquet-variant/src/builder.rs
index 3a8f7af6a077..fe3090f70b8f 100644
--- a/parquet-variant/src/builder.rs
+++ b/parquet-variant/src/builder.rs
@@ -756,6 +756,44 @@ impl<'a, 'b> ObjectBuilder<'a, 'b> {
     }
 }
 
+/// Trait that abstracts functionality from Variant fconstruction implementations, namely
+/// `VariantBuilder`, `ListBuilder` and `ObjectFieldBuilder` to minimize code duplication.
+pub(crate) trait VariantBuilderExt<'m, 'v> {
+    fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>);
+
+    fn new_list(&mut self) -> ListBuilder;
+
+    fn new_object(&mut self) -> ObjectBuilder;
+}
+
+impl<'m, 'v> VariantBuilderExt<'m, 'v> for ListBuilder<'_> {
+    fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>) {
+        self.append_value(value);
+    }
+
+    fn new_list(&mut self) -> ListBuilder {
+        self.new_list()
+    }
+
+    fn new_object(&mut self) -> ObjectBuilder {
+        self.new_object()
+    }
+}
+
+impl<'m, 'v> VariantBuilderExt<'m, 'v> for VariantBuilder {
+    fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>) {
+        self.append_value(value);
+    }
+
+    fn new_list(&mut self) -> ListBuilder {
+        self.new_list()
+    }
+
+    fn new_object(&mut self) -> ObjectBuilder {
+        self.new_object()
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
diff --git a/parquet-variant/src/from_json.rs b/parquet-variant/src/from_json.rs
new file mode 100644
index 000000000000..00d205f38584
--- /dev/null
+++ b/parquet-variant/src/from_json.rs
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Module for parsing JSON strings as Variant
+
+use crate::{ListBuilder, ObjectBuilder, Variant, VariantBuilder, VariantBuilderExt};
+use arrow_schema::ArrowError;
+use serde_json::{Number, Value};
+
+/// Converts a JSON string to Variant using [`VariantBuilder`]. The resulting `value` and `metadata`
+/// buffers can be extracted using `builder.finish()`
+///
+/// # Arguments
+/// * `json` - The JSON string to parse as Variant.
+/// * `variant_builder` - Object of type `VariantBuilder` used to build the vatiant from the JSON
+///   string
+///
+/// # Returns
+///
+/// * `Ok(())` if successful
+/// * `Err` with error details if the conversion fails
+///
+/// ```rust
+/// # use parquet_variant::{
+/// json_to_variant, variant_to_json, variant_to_json_string, variant_to_json_value, VariantBuilder
+/// };
+///
+/// let mut variant_builder = VariantBuilder::new();
+/// let person_string = "{\"name\":\"Alice\", \"age\":30, ".to_string()
+/// + "\"email\":\"alice@example.com\", \"is_active\": true, \"score\": 95.7,"
+/// + "\"additional_info\": null}";
+/// json_to_variant(&person_string, &mut variant_builder)?;
+///
+/// let (metadata, value) = variant_builder.finish();
+///
+/// let variant = parquet_variant::Variant::try_new(&metadata, &value)?;
+///
+/// let json_result = variant_to_json_string(&variant)?;
+/// let json_value = variant_to_json_value(&variant)?;
+///
+/// let mut buffer = Vec::new();
+/// variant_to_json(&mut buffer, &variant)?;
+/// let buffer_result = String::from_utf8(buffer)?;
+/// assert_eq!(json_result, "{\"additional_info\":null,\"age\":30,".to_string() +
+/// "\"email\":\"alice@example.com\",\"is_active\":true,\"name\":\"Alice\",\"score\":95.7}");
+/// assert_eq!(json_result, buffer_result);
+/// assert_eq!(json_result, serde_json::to_string(&json_value)?);
+/// # Ok::<(), Box<dyn std::error::Error>>(())
+/// ```
+pub fn json_to_variant(json: &str, builder: &mut VariantBuilder) -> Result<(), ArrowError> {
+    let json: Value = serde_json::from_str(json)
+        .map_err(|e| ArrowError::InvalidArgumentError(format!("JSON format error: {e}")))?;
+
+    build_json(&json, builder)?;
+    Ok(())
+}
+
+fn build_json(json: &Value, builder: &mut VariantBuilder) -> Result<(), ArrowError> {
+    append_json(json, builder)?;
+    Ok(())
+}
+
+fn variant_from_number<'m, 'v>(n: &Number) -> Result<Variant<'m, 'v>, ArrowError> {
+    if let Some(i) = n.as_i64() {
+        // Find minimum Integer width to fit
+        if i as i8 as i64 == i {
+            Ok((i as i8).into())
+        } else if i as i16 as i64 == i {
+            Ok((i as i16).into())
+        } else if i as i32 as i64 == i {
+            Ok((i as i32).into())
+        } else {
+            Ok(i.into())
+        }
+    } else {
+        // Todo: Try decimal once we implement custom JSON parsing where we have access to strings
+        // Try double - currently json_to_variant does not produce decimal
+        match n.as_f64() {
+            Some(f) => return Ok(f.into()),
+            None => Err(ArrowError::InvalidArgumentError(format!(
+                "Failed to parse {n} as number",
+            ))),
+        }?
+    }
+}
+
+fn append_json<'m, 'v>(
+    json: &'v Value,
+    builder: &mut impl VariantBuilderExt<'m, 'v>,
+) -> Result<(), ArrowError> {
+    match json {
+        Value::Null => builder.append_value(Variant::Null),
+        Value::Bool(b) => builder.append_value(*b),
+        Value::Number(n) => {
+            builder.append_value(variant_from_number(n)?);
+        }
+        Value::String(s) => builder.append_value(s.as_str()),
+        Value::Array(arr) => {
+            let mut list_builder = builder.new_list();
+            for val in arr {
+                append_json(val, &mut list_builder)?;
+            }
+            list_builder.finish();
+        }
+        Value::Object(obj) => {
+            let mut obj_builder = builder.new_object();
+            for (key, value) in obj.iter() {
+                let mut field_builder = ObjectFieldBuilder {
+                    key,
+                    builder: &mut obj_builder,
+                };
+                append_json(value, &mut field_builder)?;
+            }
+            obj_builder.finish()?;
+        }
+    };
+    Ok(())
+}
+
+struct ObjectFieldBuilder<'s, 'o, 'v> {
+    key: &'s str,
+    builder: &'o mut ObjectBuilder<'v, 's>,
+}
+
+impl<'m, 'v> VariantBuilderExt<'m, 'v> for ObjectFieldBuilder<'_, '_, '_> {
+    fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>) {
+        self.builder.insert(self.key, value);
+    }
+
+    fn new_list(&mut self) -> ListBuilder {
+        self.builder.new_list(self.key)
+    }
+
+    fn new_object(&mut self) -> ObjectBuilder {
+        self.builder.new_object(self.key)
+    }
+}
diff --git a/parquet-variant/src/lib.rs b/parquet-variant/src/lib.rs
index 8ce3008655d4..7dbfff52b1b5 100644
--- a/parquet-variant/src/lib.rs
+++ b/parquet-variant/src/lib.rs
@@ -33,10 +33,12 @@ mod decoder;
 mod variant;
 // TODO: dead code removal
 mod builder;
+mod from_json;
 mod to_json;
 #[allow(dead_code)]
 mod utils;
 
 pub use builder::*;
+pub use from_json::json_to_variant;
 pub use to_json::{variant_to_json, variant_to_json_string, variant_to_json_value};
 pub use variant::*;
diff --git a/parquet-variant/src/variant/decimal.rs b/parquet-variant/src/variant/decimal.rs
index 1a897d0668ab..4793d88569bf 100644
--- a/parquet-variant/src/variant/decimal.rs
+++ b/parquet-variant/src/variant/decimal.rs
@@ -86,8 +86,8 @@ pub struct VariantDecimal4 {
 }
 
 impl VariantDecimal4 {
-    const MAX_PRECISION: u8 = 9;
-    const MAX_UNSCALED_VALUE: u32 = u32::pow(10, Self::MAX_PRECISION as u32) - 1;
+    pub(crate) const MAX_PRECISION: u8 = 9;
+    pub(crate) const MAX_UNSCALED_VALUE: u32 = u32::pow(10, Self::MAX_PRECISION as u32) - 1;
 
     pub fn try_new(integer: i32, scale: u8) -> Result<Self, ArrowError> {
         decimal_try_new!(integer, scale)
@@ -136,8 +136,8 @@ pub struct VariantDecimal8 {
 }
 
 impl VariantDecimal8 {
-    const MAX_PRECISION: u8 = 18;
-    const MAX_UNSCALED_VALUE: u64 = u64::pow(10, Self::MAX_PRECISION as u32) - 1;
+    pub(crate) const MAX_PRECISION: u8 = 18;
+    pub(crate) const MAX_UNSCALED_VALUE: u64 = u64::pow(10, Self::MAX_PRECISION as u32) - 1;
 
     pub fn try_new(integer: i64, scale: u8) -> Result<Self, ArrowError> {
         decimal_try_new!(integer, scale)