Repository: arkworks-rs/algebra
Pull request #967: feat: serde-compatible adapters
Author: login: emberian
Created: 2025-04-02T00:27:33Z | Merged: 2025-04-10T08:41:29Z

## Description

Adds wrapper types that simplify using CanonicalSerialize/CanonicalDeserialize at the boundaries of `serde` types. serde-encoded-bytes is used to encode either efficient binary data for supported data formats, or else base64 fields when human readable.

The wrapper types derive all of the common traits, as well as implementing CanonicalSerialize/CanonicalDeserialize themselves, with an implementation that ignores the runtime parameters and forces validaation / compression status.

The intention is that crates that use arkworks-using crates can slap one of these wrappers around their fields at an integration interface, or use `serde_with` for semver-compatible porting. I've done a trial integration in one of my projects, and it simplifies things considerably. 

Follow-up on https://github.com/arkworks-rs/algebra/pull/506#issuecomment-1911642960

---

- [ ] Wrote unit tests: no real functionality to test



Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3366d9300..5db268fb2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,7 @@
 
 - Consolidated logic into `bitreverse_permutation_in_place` and made it public.
 - Remove redundant type constraints from `Pairing::G1Prepared`.
+- (`ark-serialize`) Add serde-compatible wrapper types `CompressedChecked<T>`, `CompressedUnchecked<T>`, `UncompressedChecked<T>`, `UncompressedUnchecked<T>`.
 
 ### Breaking changes
 
diff --git a/Cargo.toml b/Cargo.toml
index 9db3f9cbf..acecf7d12 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -169,6 +169,8 @@ rayon = "1"
 serde = "1.0"
 serde_derive = "1.0"
 serde_json = "1.0"
+serde-encoded-bytes = "0.2"
+serde_with = "3.12"
 sha2 = { version = "0.10", default-features = false }
 sha3 = { version = "0.10", default-features = false }
 blake2 = { version = "0.10", default-features = false }
diff --git a/serialize/Cargo.toml b/serialize/Cargo.toml
index f8db97bbb..a43334625 100644
--- a/serialize/Cargo.toml
+++ b/serialize/Cargo.toml
@@ -25,11 +25,15 @@ arrayvec.workspace = true
 digest.workspace = true
 num-bigint.workspace = true
 rayon = { workspace = true, optional = true }
+serde = { workspace = true, optional = true }
+serde-encoded-bytes = { workspace = true, optional = true, features = ["base64"] }
+serde_with = { workspace = true, optional = true, default-features = false }
 
 [dev-dependencies]
 sha2.workspace = true
 sha3.workspace = true
 blake2.workspace = true
+serde_json.workspace = true
 ark-test-curves = { workspace = true, default-features = false, features = [
     "bls12_381_curve",
 ] }
@@ -38,5 +42,7 @@ ark-test-curves = { workspace = true, default-features = false, features = [
 [features]
 default = []
 parallel = ["rayon"]
-std = ["ark-std/std"]
+std = ["ark-std/std", "serde_with?/std"]
 derive = ["ark-serialize-derive"]
+serde = ["dep:serde", "dep:serde-encoded-bytes", "dep:serde_with"]
+serde_with = ["serde", "dep:serde_with"]
\ No newline at end of file
diff --git a/serialize/README.md b/serialize/README.md
index d38517240..940400461 100644
--- a/serialize/README.md
+++ b/serialize/README.md
@@ -126,3 +126,64 @@ impl Valid for MyStruct {
     }
 }
 ```
+
+### `serde`-compatibility
+
+Four wrapper types (`CompressedChecked<T>`, `UncompressedChecked<T>`, `CompressedUnchecked<T>`, `UncompressedUnchecked<T>`) exist with two features: first, a type-level configuration of the de/serialization modes, and support for the [`serde`](https://serde.rs) framework. The easiest way to use these is with the `from` and `into` annotations on a container type already implementing `Canonical*`:
+
+```rust,ignore
+use ark_serialize::{CanonicalSerialize, CanonicalDeserialize, CompressedChecked};
+use ark_test_curves::bls12_381::{G1Affine, G2Affine, Fr};
+use serde::{Serialize, Deserialize};
+
+#[derive(Clone, Debug, PartialEq, Eq, CanonicalSerialize, CanonicalDeserialize)]
+pub struct Proof(pub G1Affine);
+
+#[derive(Clone, Debug, PartialEq, Eq, CanonicalSerialize, CanonicalDeserialize)]
+#[derive(Serialize, Deserialize)]
+#[serde(from = "CompressedChecked<Self>", into = "CompressedChecked<Self>")]
+pub struct Signature {
+    /// Aggregated BLS signature σ' = H(m)^aSK.
+    pub agg_sig_prime: G2Affine,
+    /// The threshold for the signature.
+    pub threshold: Fr,
+    /// The SNARK proof π.
+    pub proof: Proof,
+}
+
+impl From<CompressedChecked<Signature>> for Signature {
+    fn from(sig: CompressedChecked<Signature>) -> Self {
+        sig.0
+    }
+}
+```
+
+This type can now be used with either serialization framework.
+
+When the container type isn't suitable for `#[serde(from/into)]`, `serde_with` conversions can be used instead of pushing the wrapper types into the field values. This prevents the conversion structures from polluting the real datastructure (eg having to add dummy constructors and strip them out of collections), minimizing the impact of adding serde support.
+
+If `serde_with` is not desired, the less powerful `vec_*` mods provide support just for the common case of sequences. Without this converter, a `CompressedChecked<Vec<T>>` gets serialized as a single blob, as opposed to one blob per element, which is probably more intuitive. The alternative, trying to store `Vec<CompressedCheck<T>>`, causes invasive API pollution.
+
+```rust,ignore
+use ark_serialize::CompressedChecked;
+use serde::Serialize;
+use ark_test_curves::bls12_381::{G1Affine, G2Affine, Fr};
+
+#[serde_with::serde_as]
+#[derive(Clone, Debug, Serialize)]
+pub struct GlobalData {
+    #[serde_as(as = "CompressedChecked<UniversalParams<Curve>>")]
+    pub params: UniversalParams<Curve>,
+    pub degree_max: usize,
+    /// Used for domain separation in lockstitch
+    pub domain: String,
+    #[serde_as(as = "Vec<CompressedChecked<DensePolynomial<F>>>")]
+    pub(crate) lagrange_polynomials: Vec<DensePolynomial<F>>,
+    #[serde(with = "ark_serialize::vec_compressed_checked")]
+    pub(crate) lagrange_coms_g1: Vec<G1Affine>,
+    #[serde(with = "ark_serialize::vec_compressed_checked")]
+    pub(crate) lagrange_coms_g2: Vec<G2Affine>,
+    #[serde(skip)]
+    pub(crate) lockstitch: lockstitch::Protocol,
+}
+```
diff --git a/serialize/src/impls.rs b/serialize/src/impls.rs
index b233b620a..f6aba3c6a 100644
--- a/serialize/src/impls.rs
+++ b/serialize/src/impls.rs
@@ -980,3 +980,31 @@ where
             .collect()
     }
 }
+
+impl<T: CanonicalSerialize> CanonicalSerialize for &T {
+    fn serialize_with_mode<W: Write>(
+        &self,
+        writer: W,
+        compress: Compress,
+    ) -> Result<(), SerializationError> {
+        (*self).serialize_with_mode(writer, compress)
+    }
+
+    fn serialized_size(&self, compress: Compress) -> usize {
+        (*self).serialized_size(compress)
+    }
+}
+
+impl<T: CanonicalSerialize> CanonicalSerialize for &mut T {
+    fn serialize_with_mode<W: Write>(
+        &self,
+        writer: W,
+        compress: Compress,
+    ) -> Result<(), SerializationError> {
+        (**self).serialize_with_mode(writer, compress)
+    }
+
+    fn serialized_size(&self, compress: Compress) -> usize {
+        (**self).serialized_size(compress)
+    }
+}
diff --git a/serialize/src/lib.rs b/serialize/src/lib.rs
index 4c0b41ce0..e49b1e2f2 100644
--- a/serialize/src/lib.rs
+++ b/serialize/src/lib.rs
@@ -12,10 +12,13 @@ mod error;
 mod flags;
 mod impls;
 
+mod serde;
+
 pub use ark_std::io::{Read, Write};
 
 pub use error::*;
 pub use flags::*;
+pub use serde::*;
 
 #[cfg(test)]
 mod test;
diff --git a/serialize/src/serde.rs b/serialize/src/serde.rs
new file mode 100644
index 000000000..19a831a27
--- /dev/null
+++ b/serialize/src/serde.rs
@@ -0,0 +1,254 @@
+use ark_std::ops::{Deref, DerefMut};
+
+#[cfg(feature = "serde")]
+use ark_std::string::ToString;
+
+use crate::*;
+
+/// A serde-compatible wrapper that forces compression and skips validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct CompressedUnchecked<T>(pub T);
+
+/// A serde-compatible wrapper that skips compression and skips validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct UncompressedUnchecked<T>(pub T);
+
+/// A serde-compatible wrapper that forces compression and validates.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct CompressedChecked<T>(pub T);
+
+/// A serde-compatible wrapper that skips compression and forces validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct UncompressedChecked<T>(pub T);
+
+macro_rules! impl_ops {
+    ($type:ty, $cons:expr) => {
+        impl<T> Deref for $type {
+            type Target = T;
+
+            fn deref(&self) -> &Self::Target {
+                &self.0
+            }
+        }
+
+        impl<T> DerefMut for $type {
+            fn deref_mut(&mut self) -> &mut Self::Target {
+                &mut self.0
+            }
+        }
+
+        impl<T> From<T> for $type {
+            fn from(value: T) -> $type {
+                $cons(value)
+            }
+        }
+    };
+}
+
+macro_rules! impl_serde {
+    ($type:ty, $constr:ident, $compress:expr, $validate:expr, $vecmod:ident) => {
+        #[cfg(feature = "serde")]
+        impl<T> ::serde::Serialize for $type
+        where
+            T: CanonicalSerialize,
+        {
+            fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+            where
+                S: ::serde::Serializer,
+            {
+                let mut buf = ark_std::vec![];
+                self.0
+                    .serialize_with_mode(&mut buf, $compress)
+                    .map_err(|e| ::serde::ser::Error::custom(e.to_string()))?;
+                serde_encoded_bytes::SliceLike::<serde_encoded_bytes::Base64>::serialize(
+                    &buf, serializer,
+                )
+            }
+        }
+
+        #[cfg(feature = "serde")]
+        impl<'de, T> ::serde::Deserialize<'de> for $type
+        where
+            T: CanonicalDeserialize,
+        {
+            fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+            where
+                D: ::serde::Deserializer<'de>,
+            {
+                let buf: ark_std::vec::Vec<u8> = <serde_encoded_bytes::SliceLike<
+                    serde_encoded_bytes::Base64,
+                >>::deserialize(deserializer)?;
+                let t = T::deserialize_with_mode(&buf[..], $compress, $validate)
+                    .map_err(|e| ::serde::de::Error::custom(e.to_string()))?;
+                Ok(Self(t))
+            }
+        }
+
+        #[cfg(feature = "serde_with")]
+        impl<T> serde_with::SerializeAs<T> for $type
+        where
+            T: CanonicalSerialize,
+        {
+            fn serialize_as<S>(value: &T, serializer: S) -> Result<S::Ok, S::Error>
+            where
+                S: ::serde::Serializer,
+            {
+                use ::serde::Serialize;
+                $constr(value).serialize(serializer)
+            }
+        }
+
+        #[cfg(feature = "serde_with")]
+        impl<'de, T> serde_with::DeserializeAs<'de, T> for $type
+        where
+            T: CanonicalDeserialize,
+        {
+            fn deserialize_as<D>(deserializer: D) -> Result<T, D::Error>
+            where
+                D: ::serde::Deserializer<'de>,
+            {
+                use ::serde::Deserialize;
+                let val: Self = Self::deserialize(deserializer)?;
+                Ok(val.0)
+            }
+        }
+
+        #[cfg(feature = "serde")]
+        pub mod $vecmod {
+            use crate::{CanonicalDeserialize, CanonicalSerialize, CompressedChecked};
+            use ::serde::ser::SerializeSeq;
+            use ::serde::{Deserializer, Serializer};
+            use ark_std::fmt;
+            use ark_std::vec::Vec;
+
+            pub fn serialize<S, T>(
+                value: &impl AsRef<[T]>,
+                serializer: S,
+            ) -> Result<S::Ok, S::Error>
+            where
+                S: Serializer,
+                T: CanonicalSerialize,
+            {
+                let values = value.as_ref();
+                let mut seq = serializer.serialize_seq(Some(values.len()))?;
+                for val in values {
+                    seq.serialize_element(&CompressedChecked(val))?;
+                }
+                seq.end()
+            }
+
+            pub fn deserialize<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
+            where
+                D: Deserializer<'de>,
+                T: CanonicalDeserialize,
+            {
+                struct VecCanonicalVisitor<T> {
+                    accum: Vec<T>,
+                }
+                impl<'de, T> serde::de::Visitor<'de> for VecCanonicalVisitor<T>
+                where
+                    T: CanonicalDeserialize,
+                {
+                    type Value = Vec<T>;
+
+                    fn expecting(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
+                        write!(
+                            formatter,
+                            "a sequence of CompressedChecked<{}>",
+                            ark_std::any::type_name::<T>()
+                        )
+                    }
+
+                    fn visit_seq<A>(mut self, mut seq: A) -> Result<Self::Value, A::Error>
+                    where
+                        A: serde::de::SeqAccess<'de>,
+                    {
+                        while let Some(elt) = seq.next_element::<CompressedChecked<T>>()? {
+                            self.accum.push(elt.0);
+                        }
+                        Ok(self.accum)
+                    }
+                }
+                deserializer.deserialize_seq(VecCanonicalVisitor { accum: Vec::new() })
+            }
+        }
+    };
+}
+
+macro_rules! impl_canonical {
+    ($type:ty, $compress:expr, $validate:expr) => {
+        impl<T: CanonicalSerialize> CanonicalSerialize for $type {
+            fn serialize_with_mode<W: Write>(
+                &self,
+                writer: W,
+                _compress: Compress,
+            ) -> Result<(), SerializationError> {
+                match $compress {
+                    Compress::Yes => self.0.serialize_compressed(writer),
+                    Compress::No => self.0.serialize_uncompressed(writer),
+                }
+            }
+
+            fn serialized_size(&self, _compress: Compress) -> usize {
+                self.0.serialized_size($compress)
+            }
+        }
+
+        impl<T: Valid> Valid for $type {
+            fn check(&self) -> Result<(), SerializationError> {
+                self.0.check()
+            }
+        }
+
+        impl<T: CanonicalDeserialize> CanonicalDeserialize for $type {
+            fn deserialize_with_mode<R: Read>(
+                reader: R,
+                _compress: Compress,
+                _validate: Validate,
+            ) -> Result<Self, SerializationError> {
+                Ok(Self(T::deserialize_with_mode(
+                    reader, $compress, $validate,
+                )?))
+            }
+        }
+    };
+}
+
+impl_ops!(CompressedUnchecked<T>, CompressedUnchecked);
+impl_ops!(UncompressedUnchecked<T>, UncompressedUnchecked);
+impl_ops!(CompressedChecked<T>, CompressedChecked);
+impl_ops!(UncompressedChecked<T>, UncompressedChecked);
+
+impl_serde!(
+    CompressedUnchecked<T>,
+    CompressedUnchecked,
+    Compress::Yes,
+    Validate::No,
+    vec_compressed_unchecked
+);
+impl_serde!(
+    UncompressedUnchecked<T>,
+    UncompressedUnchecked,
+    Compress::No,
+    Validate::No,
+    vec_uncompressed_unchecked
+);
+impl_serde!(
+    CompressedChecked<T>,
+    CompressedChecked,
+    Compress::Yes,
+    Validate::Yes,
+    vec_compressed_checked
+);
+impl_serde!(
+    UncompressedChecked<T>,
+    UncompressedChecked,
+    Compress::No,
+    Validate::Yes,
+    vec_uncompressed_checked
+);
+
+impl_canonical!(CompressedUnchecked<T>, Compress::Yes, Validate::No);
+impl_canonical!(UncompressedUnchecked<T>, Compress::No, Validate::No);
+impl_canonical!(CompressedChecked<T>, Compress::Yes, Validate::Yes);
+impl_canonical!(UncompressedChecked<T>, Compress::No, Validate::Yes);
diff --git a/serialize/src/test.rs b/serialize/src/test.rs
index 72f20fa13..59af4ea0b 100644
--- a/serialize/src/test.rs
+++ b/serialize/src/test.rs
@@ -57,6 +57,33 @@ impl CanonicalDeserialize for Dummy {
 fn test_serialize<T: PartialEq + core::fmt::Debug + CanonicalSerialize + CanonicalDeserialize>(
     data: T,
 ) {
+    #[cfg(feature = "serde")]
+    let serde_ser = |compress: Compress, validate: Validate| -> Result<String, serde_json::Error> {
+        match (compress, validate) {
+            (Compress::Yes, Validate::Yes) => serde_json::to_string(&CompressedChecked(&data)),
+            (Compress::Yes, Validate::No) => serde_json::to_string(&CompressedUnchecked(&data)),
+            (Compress::No, Validate::Yes) => serde_json::to_string(&UncompressedChecked(&data)),
+            (Compress::No, Validate::No) => serde_json::to_string(&UncompressedUnchecked(&data)),
+        }
+    };
+    #[cfg(feature = "serde")]
+    let serde_de =
+        |compress: Compress, validate: Validate, input: &str| -> Result<T, serde_json::Error> {
+            Ok(match (compress, validate) {
+                (Compress::Yes, Validate::Yes) => {
+                    serde_json::from_str::<CompressedChecked<T>>(input)?.0
+                },
+                (Compress::Yes, Validate::No) => {
+                    serde_json::from_str::<CompressedUnchecked<T>>(input)?.0
+                },
+                (Compress::No, Validate::Yes) => {
+                    serde_json::from_str::<UncompressedChecked<T>>(input)?.0
+                },
+                (Compress::No, Validate::No) => {
+                    serde_json::from_str::<UncompressedUnchecked<T>>(input)?.0
+                },
+            })
+        };
     for compress in [Compress::Yes, Compress::No] {
         for validate in [Validate::Yes, Validate::No] {
             let mut serialized = vec![0; data.serialized_size(compress)];
@@ -64,6 +91,12 @@ fn test_serialize<T: PartialEq + core::fmt::Debug + CanonicalSerialize + Canonic
                 .unwrap();
             let de = T::deserialize_with_mode(&serialized[..], compress, validate).unwrap();
             assert_eq!(data, de);
+            #[cfg(feature = "serde")]
+            {
+                let serde_serialized = serde_ser(compress, validate).unwrap();
+                let serde_de = serde_de(compress, validate, &serde_serialized).unwrap();
+                assert_eq!(data, serde_de);
+            }
         }
     }
 }


Reviews:
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:32:08Z
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:32:59Z
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:35:59Z
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:37:52Z
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:38:41Z
user: login: emberian
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T07:15:09Z
user: login: Pratyush
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-04T19:38:47Z
user: login: Pratyush
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-10T08:30:31Z
user: login: burdges
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-07-21T10:56:53Z
user: login: emberian
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-08-08T06:35:10Z
user: login: burdges
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-08-09T09:03:06Z

Review comments:
diff_hunk: @@ -38,5 +41,7 @@ ark-test-curves = { workspace = true, default-features = false, features = [
 [features]
 default = []
 parallel = ["rayon"]
-std = ["ark-std/std"]
+std = ["ark-std/std", "serde_with/std"]
path: serialize/Cargo.toml
user: login: Pratyush
body: I'd prefer it not be a default feature, as then every dependent crate would immediately depend on `serde`.
created_at: 2025-04-02T06:32:08Z
author_association: MEMBER
diff_hunk: @@ -12,10 +12,13 @@ mod error;
 mod flags;
 mod impls;
 
+mod serde;
path: serialize/src/lib.rs
user: login: Pratyush
body: Or should this be `serde_with`?

```suggestion
#[cfg(feature = "serde")]
mod serde;
```
created_at: 2025-04-02T06:32:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,254 @@
+use ark_std::ops::{Deref, DerefMut};
+
+#[cfg(feature = "serde")]
+use ark_std::string::ToString;
+
+use crate::*;
+
+/// A serde-compatible wrapper that forces compression and skips validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct CompressedUnchecked<T>(pub T);
+
+/// A serde-compatible wrapper that skips compression and skips validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct UncompressedUnchecked<T>(pub T);
+
+/// A serde-compatible wrapper that forces compression and validates.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct CompressedChecked<T>(pub T);
+
+/// A serde-compatible wrapper that skips compression and forces validation.
+#[derive(Copy, Clone, Eq, PartialEq, Ord, PartialOrd, Hash, Debug, Default)]
+pub struct UncompressedChecked<T>(pub T);
+
+macro_rules! impl_ops {
+    ($type:ty, $cons:expr) => {
+        impl<T> Deref for $type {
+            type Target = T;
+
+            fn deref(&self) -> &Self::Target {
+                &self.0
+            }
+        }
+
+        impl<T> DerefMut for $type {
+            fn deref_mut(&mut self) -> &mut Self::Target {
+                &mut self.0
+            }
+        }
+
+        impl<T> From<T> for $type {
+            fn from(value: T) -> $type {
+                $cons(value)
+            }
+        }
+
+        impl<T: Valid> Valid for $type {
+            fn check(&self) -> Result<(), SerializationError> {
+                self.0.check()
+            }
+        }
path: serialize/src/serde.rs
user: login: Pratyush
body: Probably this should move into `impl_canonical` as it is logically related to `CanonicalDeserialize`
created_at: 2025-04-02T06:35:59Z
author_association: MEMBER
diff_hunk: @@ -38,5 +41,7 @@ ark-test-curves = { workspace = true, default-features = false, features = [
 [features]
 default = []
 parallel = ["rayon"]
-std = ["ark-std/std"]
+std = ["ark-std/std", "serde_with/std"]
path: serialize/Cargo.toml
user: login: Pratyush
body: Oh nvm, I see what's going on here. You might need to use the optional feature syntax here.
```suggestion
std = ["ark-std/std", "serde_with?/std"]
```
created_at: 2025-04-02T06:37:52Z
author_association: MEMBER
diff_hunk: @@ -38,5 +41,7 @@ ark-test-curves = { workspace = true, default-features = false, features = [
 [features]
 default = []
 parallel = ["rayon"]
-std = ["ark-std/std"]
+std = ["ark-std/std", "serde_with/std"]
path: serialize/Cargo.toml
user: login: Pratyush
body: See here: https://doc.rust-lang.org/cargo/reference/features.html#dependency-features
created_at: 2025-04-02T06:38:41Z
author_association: MEMBER
diff_hunk: @@ -12,10 +12,13 @@ mod error;
 mod flags;
 mod impls;
 
+mod serde;
path: serialize/src/lib.rs
user: login: emberian
body: i wonder if it's a good idea to keep the marker types unconditionally available, because they have minor utility for type-driven serialozation mode customization? seems rarely useful or wanted though.

as a zero-cost annotation they could foreseeably be used in public APIs.

but i'm not sure it helps in practice
created_at: 2025-04-02T07:15:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,10 +12,13 @@ mod error;
 mod flags;
 mod impls;
 
+mod serde;
path: serialize/src/lib.rs
user: login: Pratyush
body: Hm that's a good idea. Sure, we can keep them available.
created_at: 2025-04-04T19:38:47Z
author_association: MEMBER
diff_hunk: @@ -25,11 +25,15 @@ arrayvec.workspace = true
 digest.workspace = true
 num-bigint.workspace = true
 rayon = { workspace = true, optional = true }
+serde = { workspace = true, optional = true }
+serde-encoded-bytes = { workspace = true, optional = true, features = ["base64"] }
+serde_with = { workspace = true, optional = true, default-features = false }
 
 [dev-dependencies]
 sha2.workspace = true
 sha3.workspace = true
 blake2.workspace = true
+serde_json.workspace = true
path: serialize/Cargo.toml
user: login: burdges
body: This should not be on by default.

Ideally serde might not even be an optinal dependency, and not use features, but only be a dependency of a seperate crate, not sure how much difference that'd make really though. 
created_at: 2025-07-21T10:56:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -25,11 +25,15 @@ arrayvec.workspace = true
 digest.workspace = true
 num-bigint.workspace = true
 rayon = { workspace = true, optional = true }
+serde = { workspace = true, optional = true }
+serde-encoded-bytes = { workspace = true, optional = true, features = ["base64"] }
+serde_with = { workspace = true, optional = true, default-features = false }
 
 [dev-dependencies]
 sha2.workspace = true
 sha3.workspace = true
 blake2.workspace = true
+serde_json.workspace = true
path: serialize/Cargo.toml
user: login: emberian
body: @burdges the tests use serde_json; doesn't that require a dev-dependencies be active by default?
created_at: 2025-08-08T06:35:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -25,11 +25,15 @@ arrayvec.workspace = true
 digest.workspace = true
 num-bigint.workspace = true
 rayon = { workspace = true, optional = true }
+serde = { workspace = true, optional = true }
+serde-encoded-bytes = { workspace = true, optional = true, features = ["base64"] }
+serde_with = { workspace = true, optional = true, default-features = false }
 
 [dev-dependencies]
 sha2.workspace = true
 sha3.workspace = true
 blake2.workspace = true
+serde_json.workspace = true
path: serialize/Cargo.toml
user: login: burdges
body: I suppose okay, maybe it could do so only in CI here?

The concerns over serde is first that it shipped a binary that runs during builds once before, but this is not currently the case, and second that at least one of the maintainers supposedly works for Palantir. 
created_at: 2025-08-09T09:03:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: emberian
created_at: 2025-04-02T00:28:10Z
body: hello @mmaker, i made some progress towards the comment you left, care to take a look?
author_association: CONTRIBUTOR
user: login: emberian
created_at: 2025-04-02T01:02:15Z
body: With the additional `From` impl, a usage looks like:

```rs
/// Final Aggregated Signature.
#[derive(Clone, Debug, PartialEq, Eq, CanonicalSerialize, CanonicalDeserialize)]
#[derive(Serialize,Deserialize)]
#[serde(from = "CompressedChecked<Self>", into = "CompressedChecked<Self>")]
pub struct Signature {
    /// Aggregated BLS signature σ' = H(m)^aSK.
    pub agg_sig_prime: G2,
    /// The threshold for the signature.
    pub threshold: F,
    /// The SNARK proof π.
    pub proof: Proof,
}

impl From<CompressedChecked<Signature>> for Signature {
    fn from(sig: CompressedChecked<Signature>) -> Self {
        sig.0
    }
}
```
author_association: CONTRIBUTOR
user: login: emberian
created_at: 2025-04-02T05:39:10Z
body: Here is a more detailed example showing the latest integration with `serde_with`. When the container type isn't suitable for `#[serde(from/into)]`, these conversions can be used instead of pushing the wrapper types into the field values. This prevents the conversion structures from polluting the real datastructure (eg having to add dummy constructors and strip them out of collections), minimizing the impact of adding serde support.

If `serde_with` is not desired, the less powerful `vec_*` mods provide support just for the common case of sequences. Without this converter, a `CompressedChecked<Vec<T>>` gets serialized as a single blob, as opposed to one blob per element, which is probably more intuitive. The alternative, trying to store `Vec<CompressedCheck<T>>`, causes invasive pollution.

```rs

/// Global data for a hints setup.
///
/// Contains the KZG parameters and the cache of precomputed Lagrange polynomials and commitments.
#[serde_with::serde_as]
#[derive(Clone, Debug, Serialize)]
pub struct GlobalData {
    #[serde_as(as = "CompressedChecked<UniversalParams<Curve>>")]
    pub params: UniversalParams<Curve>,
    pub degree_max: usize,
    /// Used for domain separation in lockstitch
    pub domain: String,
    #[serde_as(as = "Vec<CompressedChecked<DensePolynomial<F>>>")]
    pub(crate) lagrange_polynomials: Vec<DensePolynomial<F>>,
    #[serde(with = "ark_serialize::vec_compressed_checked")]
    pub(crate) lagrange_coms_g1: Vec<G1>,
    #[serde(with = "ark_serialize::vec_compressed_checked")]
    pub(crate) lagrange_coms_g2: Vec<G2>,
    #[serde(skip)]
    pub(crate) lockstitch: lockstitch::Protocol,
}
```
author_association: CONTRIBUTOR
user: login: Pratyush
created_at: 2025-04-02T06:42:54Z
body: Hi @emberian,

Thank you for the PR! This is a great implementation. I just had a couple of requests. Could you add your code examples of how to use `#[serde(from/into)]` and `#[serde_with]` as documentation in the code? Additionally, if you could add small unit tests that do a roundtrip serde serialization/deserialization (e.g., with serde-json or any other serde package), that would be great
author_association: MEMBER
user: login: Pratyush
created_at: 2025-04-04T19:39:24Z
body: Hey @emberian ! The PR looks good modulo the doc examples.
author_association: MEMBER
user: login: emberian
created_at: 2025-04-07T03:46:56Z
body: hi @Pratyush i think this is ready for merging :)

I ended up having to `ignore` the new doc example code because it is not possible to enable a crate `feature` just for the docs.
author_association: CONTRIBUTOR