Repository: risingwavelabs/risingwave (language: python)

Issue:
fix(source): resolve avro `Ref` during `avro_schema_to_column_descs` without hack
I hereby agree to the terms of the [RisingWave Labs, Inc. Contributor License Agreement](https://gist.github.com/TennyZhuang/f00be7f16996ea48effb049aa7be4d66#file-rw_cla).

## What's changed and what's your intention?

Part of #17020. `Ref` in avro used to be supported by a hack we added in our apache_avro fork. This PR avoids the hack (replacing `Ref` with their referee in-place, resulting in an invalid tree containing duplicate definitions). The hack fails to work when there is `Ref` inside `Ref` - resulting in either unresolved `Ref` or infinite recursion.

This PR only corrects part of the usage - in `avro_schema_to_column_descs` to derive RisingWave column data types from avro schema. There will be a follow-up to correct the usage in `convert_to_datum`/`AvroAccess`. Without the latter part, simple data types like `int` (inside a `Ref` in another `Ref`) can already be supported.

Instead of building the expanded-yet-invalid tree as in the hack, this solution passes a `NamesRef` obtained from the root. Its complexity of having an associated lifetime is easy to deal with in this context. (To contrast, `prost_reflect` for protobuf builds the tree with `Arc` - no lifetime and no duplication. But `apache_avro::Schema` does not use `Arc`.)

The circular reference rejection logic is same as #10499 for protobuf. It can be DRY'ed with error message improved later.

This is intended to be part of v2.2 and NOT cherry-picked into earlier versions. There has been a further hack available for earlier versions that can be cherry-picked on demand - where the user is responsible for not using circular reference or an infinite recursion would happen.

<!--

**Please do not leave this empty!**

Please explain **IN DETAIL** what the changes are in this PR and why they are needed:

- Summarize your change (**mandatory**)
- How does this PR work? Need a brief introduction for the changed logic (optional)
- Describe clearly one logical change and avoid lazy messages (optional)
- Describe any limitations of the current code (optional)
- Refer to a related PR or issue link (optional)

-->

## Checklist

- [x] I have written necessary rustdoc comments
- [x] I have added necessary unit tests and integration tests
- [ ] I have added test labels as necessary. See [details](https://github.com/risingwavelabs/risingwave/blob/main/docs/developer-guide.md#ci-labels-guide).
- [ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
- [ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
- [x] All checks passed in `./risedev check` (or alias, `./risedev c`)
- [ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
<!-- To manually trigger a benchmark, please check out [Notion](https://www.notion.so/risingwave-labs/Manually-trigger-nexmark-performance-dashboard-test-b784f1eae1cf48889b2645d020b6b7d3). -->
- [ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the [details](https://github.com/risingwavelabs/risingwave/blob/main/CONTRIBUTING.md))

## Documentation

- [ ] My PR needs documentation updates. (Please use the **Release note** section below to summarize the impact on users)

## Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.


<!--
Please create a release note for your changes.

Discuss technical details in the "What's changed" section, and
focus on the impact on users in the release note.

You should also mention the environment or conditions where the impact may occur.
-->

</details>

Support Avro ref type in source

We added the support for Debezium avro, but not otherwise, which makes no sense.

A little background: In debezium, there are `after` and `before` sharing the same schema.

https://github.com/risingwavelabs/risingwave/blob/8dfae832334ae1dc0585dcae4b3071e9ee6c9b1d/src/connector/src/parser/debezium/avro_parser.rs#L83

BTW, this line is introduced in this huge refactor: https://github.com/risingwavelabs/risingwave/pull/10096, which is too confusing and scary. How does it work before?
fix(source): resolve avro `Ref` during `avro_schema_to_column_descs` without hack
I hereby agree to the terms of the [RisingWave Labs, Inc. Contributor License Agreement](https://gist.github.com/TennyZhuang/f00be7f16996ea48effb049aa7be4d66#file-rw_cla).

## What's changed and what's your intention?

Part of #17020. `Ref` in avro used to be supported by a hack we added in our apache_avro fork. This PR avoids the hack (replacing `Ref` with their referee in-place, resulting in an invalid tree containing duplicate definitions). The hack fails to work when there is `Ref` inside `Ref` - resulting in either unresolved `Ref` or infinite recursion.

This PR only corrects part of the usage - in `avro_schema_to_column_descs` to derive RisingWave column data types from avro schema. There will be a follow-up to correct the usage in `convert_to_datum`/`AvroAccess`. Without the latter part, simple data types like `int` (inside a `Ref` in another `Ref`) can already be supported.

Instead of building the expanded-yet-invalid tree as in the hack, this solution passes a `NamesRef` obtained from the root. Its complexity of having an associated lifetime is easy to deal with in this context. (To contrast, `prost_reflect` for protobuf builds the tree with `Arc` - no lifetime and no duplication. But `apache_avro::Schema` does not use `Arc`.)

The circular reference rejection logic is same as #10499 for protobuf. It can be DRY'ed with error message improved later.

This is intended to be part of v2.2 and NOT cherry-picked into earlier versions. There has been a further hack available for earlier versions that can be cherry-picked on demand - where the user is responsible for not using circular reference or an infinite recursion would happen.

<!--

**Please do not leave this empty!**

Please explain **IN DETAIL** what the changes are in this PR and why they are needed:

- Summarize your change (**mandatory**)
- How does this PR work? Need a brief introduction for the changed logic (optional)
- Describe clearly one logical change and avoid lazy messages (optional)
- Describe any limitations of the current code (optional)
- Refer to a related PR or issue link (optional)

-->

## Checklist

- [x] I have written necessary rustdoc comments
- [x] I have added necessary unit tests and integration tests
- [ ] I have added test labels as necessary. See [details](https://github.com/risingwavelabs/risingwave/blob/main/docs/developer-guide.md#ci-labels-guide).
- [ ] I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features #7934).
- [ ] My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
- [x] All checks passed in `./risedev check` (or alias, `./risedev c`)
- [ ] My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
<!-- To manually trigger a benchmark, please check out [Notion](https://www.notion.so/risingwave-labs/Manually-trigger-nexmark-performance-dashboard-test-b784f1eae1cf48889b2645d020b6b7d3). -->
- [ ] My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the [details](https://github.com/risingwavelabs/risingwave/blob/main/CONTRIBUTING.md))

## Documentation

- [ ] My PR needs documentation updates. (Please use the **Release note** section below to summarize the impact on users)

## Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.


<!--
Please create a release note for your changes.

Discuss technical details in the "What's changed" section, and
focus on the impact on users in the release note.

You should also mention the environment or conditions where the impact may occur.
-->

</details>

Sqlsmith: Sql feature generation
As we add more features, these should be tested via sqlsmith as well.

SQL Features:
- [ ] Generate other table functions: `generate_series`, `unnest`. See: https://github.com/singularity-data/risingwave/issues/4540
- [ ] Generate array_agg, jsonb_agg, jsonb_object_agg
- [ ] Generate select from System Tables
- [ ] Generate `NATURAL JOIN`
- [ ] Generate `JOIN` with `USING`
- [ ] Generate query w/o `GROUP BY`
- [ ] Randomly toggle session variables
- [ ] Generate `pow`/`exp`.
- [ ] Generate TopN by group: https://www.risingwave.dev/docs/current/sql-pattern-topn/
- [ ] Generate jsonb exprs, see: https://github.com/risingwavelabs/risingwave/issues/7714
- [ ] #7132
- [x] Generate order-by-clauses with `NULLS { FIRST | LAST }` (will be done in https://github.com/risingwavelabs/risingwave/pull/10864/commits/000cbc11cbc04f20abfd127b1bf4b3fd24e75c44)
- [ ] Generate `max`,`min`,`count` for `timestamptz`
- [ ] Generate `bit_and`, `bit_or` and `big_xor` in batch queries, and `bit_xor` in streaming queries.
- [ ] Generate array range access.
- [ ] https://github.com/risingwavelabs/risingwave/issues/10241
- [ ] Generate window functions other than `row_number`, `rank`.
- [ ] Generate `ALTER` DDL statements.\
- [ ] https://github.com/risingwavelabs/risingwave/issues/10516
- [ ] Generate temporal filter in sqlsmith


Discussion / hints:
For context, this is the error reported for the added test case if this PR was not applied:
```
ERROR:  Failed to run the query

Caused by these errors (recent errors listed first):
  1: connector error
  2: Feature is not yet implemented: Avro type: Ref { name: Name { name: "Point", namespace: None } }
No tracking issue yet. Feel free to submit a feature request at https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Ffeature&template=feature_request.yml
```

> There will be a follow-up to correct the usage in `convert_to_datum`/`AvroAccess`. Without the latter part, simple data types like `int` (inside a `Ref` in another `Ref`) can already be supported.

Both parts will be included in v2.2. In this intermediate state, `decimal` inside a `Ref` in another `Ref` would result in a parsing error and filled with null, instead of creation rejected without this first part.
resolve ref in schema is an internal API and has some bugs, we use our patched crate in Cargo.toml. Not sure @xiangjinwu includes the fix when migrating deps. 
> Not sure @xiangjinwu includes the fix when migrating deps.

Not yet.

> resolve ref in schema is an internal API and has some bugs, we use our patched crate in Cargo.toml. 

The bug is because we use the internal API the wrong way, or the API itself is poorly designed and hard to use. In short, we need to replaces almost all references to `Schema` into `ResolvedSchema` and use functions that take `schemata` (vec of schemas) rather than a single schema. I will work with @xxchan on resolving this.
The current cloning hack implementation fails to handle the following situation:
```
{
  "type": "record",
  "name": "Root",
  "fields": [
    {
      "name": "f1",
      "type": {
        "type": "record",
        "name": "Nested",
        "fields": [
          {
            "name": "f1",
            "type": {
              "type": "enum",
              "name": "Case",
              "symbols": ["A", "B", "C"]
            }
          },
          {
            "name": "f2",
            "type": "Case"
          }
        ]
      }
    },
    {
      "name": "f2",
      "type": "Nested"
    }
  ]
}
```

In `ResolvedAvroSchema::resolved_schema` the field `f2.f2` is not resolved and still a `ref`:
<details>

```
Record(
    RecordSchema {
        name: Name {
            name: "Root",
            namespace: None,
        },
        aliases: None,
        doc: None,
        fields: [
            RecordField {
                name: "f1",
                doc: None,
                aliases: None,
                default: None,
                schema: Record(
                    RecordSchema {
                        name: Name {
                            name: "Nested",
                            namespace: None,
                        },
                        aliases: None,
                        doc: None,
                        fields: [
                            RecordField {
                                name: "f1",
                                doc: None,
                                aliases: None,
                                default: None,
                                schema: Enum(
                                    EnumSchema {
                                        name: Name {
                                            name: "Case",
                                            namespace: None,
                                        },
                                        aliases: None,
                                        doc: None,
                                        symbols: [
                                            "A",
                                            "B",
                                            "C",
                                        ],
                                        default: None,
                                        attributes: {},
                                    },
                                ),
                                order: Ascending,
                                position: 0,
                                custom_attributes: {},
                            },
                            RecordField {
                                name: "f2",
                                doc: None,
                                aliases: None,
                                default: None,
                                schema: Enum(
                                    EnumSchema {
                                        name: Name {
                                            name: "Case",
                                            namespace: None,
                                        },
                                        aliases: None,
                                        doc: None,
                                        symbols: [
                                            "A",
                                            "B",
                                            "C",
                                        ],
                                        default: None,
                                        attributes: {},
                                    },
                                ),
                                order: Ascending,
                                position: 1,
                                custom_attributes: {},
                            },
                        ],
                        lookup: {
                            "f1": 0,
                            "f2": 1,
                        },
                        attributes: {},
                    },
                ),
                order: Ascending,
                position: 0,
                custom_attributes: {},
            },
            RecordField {
                name: "f2",
                doc: None,
                aliases: None,
                default: None,
                schema: Record(
                    RecordSchema {
                        name: Name {
                            name: "Nested",
                            namespace: None,
                        },
                        aliases: None,
                        doc: None,
                        fields: [
                            RecordField {
                                name: "f1",
                                doc: None,
                                aliases: None,
                                default: None,
                                schema: Enum(
                                    EnumSchema {
                                        name: Name {
                                            name: "Case",
                                            namespace: None,
                                        },
                                        aliases: None,
                                        doc: None,
                                        symbols: [
                                            "A",
                                            "B",
                                            "C",
                                        ],
                                        default: None,
                                        attributes: {},
                                    },
                                ),
                                order: Ascending,
                                position: 0,
                                custom_attributes: {},
                            },
                            RecordField {
                                name: "f2",
                                doc: None,
                                aliases: None,
                                default: None,
                                schema: Ref {
                                    name: Name {
                                        name: "Case",
                                        namespace: None,
                                    },
                                },
                                order: Ascending,
                                position: 1,
                                custom_attributes: {},
                            },
                        ],
                        lookup: {
                            "f1": 0,
                            "f2": 1,
                        },
                        attributes: {},
                    },
                ),
                order: Ascending,
                position: 1,
                custom_attributes: {},
            },
        ],
        lookup: {
            "f1": 0,
            "f2": 1,
        },
        attributes: {},
    },
)
```
</details>
For context, this is the error reported for the added test case if this PR was not applied:
```
ERROR:  Failed to run the query

Caused by these errors (recent errors listed first):
  1: connector error
  2: Feature is not yet implemented: Avro type: Ref { name: Name { name: "Point", namespace: None } }
No tracking issue yet. Feel free to submit a feature request at https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Ffeature&template=feature_request.yml
```

> There will be a follow-up to correct the usage in `convert_to_datum`/`AvroAccess`. Without the latter part, simple data types like `int` (inside a `Ref` in another `Ref`) can already be supported.

Both parts will be included in v2.2. In this intermediate state, `decimal` inside a `Ref` in another `Ref` would result in a parsing error and filled with null, instead of creation rejected without this first part.


Fix patch:
diff --git a/src/connector/codec/src/decoder/avro/mod.rs b/src/connector/codec/src/decoder/avro/mod.rs
index 9eddc7b722ae..c1905ba7517b 100644
--- a/src/connector/codec/src/decoder/avro/mod.rs
+++ b/src/connector/codec/src/decoder/avro/mod.rs
@@ -14,7 +14,7 @@
 
 mod schema;
 
-use apache_avro::schema::{DecimalSchema, UnionSchema};
+use apache_avro::schema::{DecimalSchema, NamesRef, UnionSchema};
 use apache_avro::types::{Value, ValueKind};
 use apache_avro::{Decimal as AvroDecimal, Schema};
 use chrono::Datelike;
@@ -33,28 +33,46 @@ use super::{bail_uncategorized, uncategorized, Access, AccessError, AccessResult
 use crate::decoder::avro::schema::avro_schema_to_struct_field_name;
 
 #[derive(Clone)]
-/// Options for parsing an `AvroValue` into Datum, with an optional avro schema.
+/// Options for parsing an `AvroValue` into Datum, with a root avro schema.
 pub struct AvroParseOptions<'a> {
-    /// Currently, this schema is only used for decimal.
-    ///
-    /// FIXME: In theory we should use resolved schema.
-    /// e.g., it's possible that a field is a reference to a decimal or a record containing a decimal field.
-    schema: &'a Schema,
+    /// The avro schema at root level
+    root_schema: &'a Schema,
+    /// The immutable "global" context during recursive parsing
+    inner: AvroParseOptionsInner<'a>,
+}
+
+#[derive(Clone)]
+/// Options for parsing an `AvroValue` into Datum, with names resolved from root schema.
+struct AvroParseOptionsInner<'a> {
+    /// Mapping from type names to actual schema
+    refs: NamesRef<'a>,
     /// Strict Mode
     /// If strict mode is disabled, an int64 can be parsed from an `AvroInt` (int32) value.
     relax_numeric: bool,
 }
 
 impl<'a> AvroParseOptions<'a> {
-    pub fn create(schema: &'a Schema) -> Self {
+    pub fn create(root_schema: &'a Schema) -> Self {
+        let resolved = apache_avro::schema::ResolvedSchema::try_from(root_schema)
+            .expect("avro schema is self contained");
         Self {
-            schema,
-            relax_numeric: true,
+            root_schema,
+            inner: AvroParseOptionsInner {
+                refs: resolved.get_names().clone(),
+                relax_numeric: true,
+            },
         }
     }
 }
 
-impl<'a> AvroParseOptions<'a> {
+impl<'a> AvroParseOptionsInner<'a> {
+    fn lookup_ref(&self, schema: &'a Schema) -> &'a Schema {
+        match schema {
+            Schema::Ref { name } => self.refs[name],
+            _ => schema,
+        }
+    }
+
     /// Parse an avro value into expected type.
     ///
     /// 3 kinds of type info are used to parsing:
@@ -70,6 +88,7 @@ impl<'a> AvroParseOptions<'a> {
     ///     the `DataType` will be inferred.
     fn convert_to_datum<'b>(
         &self,
+        unresolved_schema: &'a Schema,
         value: &'b Value,
         type_expected: &DataType,
     ) -> AccessResult<DatumCow<'b>>
@@ -92,18 +111,14 @@ impl<'a> AvroParseOptions<'a> {
             (_, Value::Null) => return Ok(DatumCow::NULL),
             // ---- Union (with >=2 non null variants), and nullable Union ([null, record]) -----
             (DataType::Struct(struct_type_info), Value::Union(variant, v)) => {
-                let Schema::Union(u) = self.schema else {
+                let Schema::Union(u) = self.lookup_ref(unresolved_schema) else {
                     // XXX: Is this branch actually unreachable? (if self.schema is correctly used)
                     return Err(create_error());
                 };
 
                 if let Some(inner) = get_nullable_union_inner(u) {
                     // nullable Union ([null, record])
-                    return Self {
-                        schema: inner,
-                        relax_numeric: self.relax_numeric,
-                    }
-                    .convert_to_datum(v, type_expected);
+                    return self.convert_to_datum(inner, v, type_expected);
                 }
                 let variant_schema = &u.variants()[*variant as usize];
 
@@ -120,12 +135,9 @@ impl<'a> AvroParseOptions<'a> {
                 let mut fields = Vec::with_capacity(struct_type_info.len());
                 for (field_name, field_type) in struct_type_info.iter() {
                     if field_name == expected_field_name {
-                        let datum = Self {
-                            schema: variant_schema,
-                            relax_numeric: self.relax_numeric,
-                        }
-                        .convert_to_datum(v, field_type)?
-                        .to_owned_datum();
+                        let datum = self
+                            .convert_to_datum(variant_schema, v, field_type)?
+                            .to_owned_datum();
 
                         fields.push(datum)
                     } else {
@@ -136,17 +148,13 @@ impl<'a> AvroParseOptions<'a> {
             }
             // nullable Union ([null, T])
             (_, Value::Union(_, v)) => {
-                let Schema::Union(u) = self.schema else {
+                let Schema::Union(u) = self.lookup_ref(unresolved_schema) else {
                     return Err(create_error());
                 };
                 let Some(schema) = get_nullable_union_inner(u) else {
                     return Err(create_error());
                 };
-                return Self {
-                    schema,
-                    relax_numeric: self.relax_numeric,
-                }
-                .convert_to_datum(v, type_expected);
+                return self.convert_to_datum(schema, v, type_expected);
             }
             // ---- Boolean -----
             (DataType::Boolean, Value::Boolean(b)) => (*b).into(),
@@ -168,7 +176,7 @@ impl<'a> AvroParseOptions<'a> {
             (DataType::Float64, Value::Float(i)) => (*i as f64).into(),
             // ---- Decimal -----
             (DataType::Decimal, Value::Decimal(avro_decimal)) => {
-                let (precision, scale) = match self.schema {
+                let (precision, scale) = match self.lookup_ref(unresolved_schema) {
                     Schema::Decimal(DecimalSchema {
                         precision, scale, ..
                     }) => (*precision, *scale),
@@ -254,7 +262,7 @@ impl<'a> AvroParseOptions<'a> {
             }
             // ---- Struct -----
             (DataType::Struct(struct_type_info), Value::Record(descs)) => StructValue::new({
-                let Schema::Record(record_schema) = &self.schema else {
+                let Schema::Record(record_schema) = self.lookup_ref(unresolved_schema) else {
                     return Err(create_error());
                 };
                 struct_type_info
@@ -264,12 +272,9 @@ impl<'a> AvroParseOptions<'a> {
                         if let Some(idx) = record_schema.lookup.get(field_name) {
                             let value = &descs[*idx].1;
                             let schema = &record_schema.fields[*idx].schema;
-                            Ok(Self {
-                                schema,
-                                relax_numeric: self.relax_numeric,
-                            }
-                            .convert_to_datum(value, field_type)?
-                            .to_owned_datum())
+                            Ok(self
+                                .convert_to_datum(schema, value, field_type)?
+                                .to_owned_datum())
                         } else {
                             Ok(None)
                         }
@@ -279,17 +284,13 @@ impl<'a> AvroParseOptions<'a> {
             .into(),
             // ---- List -----
             (DataType::List(item_type), Value::Array(array)) => ListValue::new({
-                let Schema::Array(element_schema) = &self.schema else {
+                let Schema::Array(element_schema) = self.lookup_ref(unresolved_schema) else {
                     return Err(create_error());
                 };
                 let schema = element_schema;
                 let mut builder = item_type.create_array_builder(array.len());
                 for v in array {
-                    let value = Self {
-                        schema,
-                        relax_numeric: self.relax_numeric,
-                    }
-                    .convert_to_datum(v, item_type)?;
+                    let value = self.convert_to_datum(schema, v, item_type)?;
                     builder.append(value);
                 }
                 builder.finish()
@@ -309,7 +310,7 @@ impl<'a> AvroParseOptions<'a> {
                 uuid.as_hyphenated().to_string().into_boxed_str().into()
             }
             (DataType::Map(map_type), Value::Map(map)) => {
-                let Schema::Map(value_schema) = &self.schema else {
+                let Schema::Map(value_schema) = self.lookup_ref(unresolved_schema) else {
                     return Err(create_error());
                 };
                 let schema = value_schema;
@@ -325,12 +326,9 @@ impl<'a> AvroParseOptions<'a> {
                 // in tests. We might consider removing this, or make all MapValue sorted
                 // in the future.
                 for (k, v) in map.iter().sorted_by_key(|(k, _v)| *k) {
-                    let value_datum = Self {
-                        schema,
-                        relax_numeric: self.relax_numeric,
-                    }
-                    .convert_to_datum(v, map_type.value())?
-                    .to_owned_datum();
+                    let value_datum = self
+                        .convert_to_datum(schema, v, map_type.value())?
+                        .to_owned_datum();
                     builder.append(
                         StructValue::new(vec![Some(k.as_str().into()), value_datum])
                             .to_owned_datum(),
@@ -352,15 +350,18 @@ pub struct AvroAccess<'a> {
 }
 
 impl<'a> AvroAccess<'a> {
-    pub fn new(value: &'a Value, options: AvroParseOptions<'a>) -> Self {
-        Self { value, options }
+    pub fn new(root_value: &'a Value, options: AvroParseOptions<'a>) -> Self {
+        Self {
+            value: root_value,
+            options,
+        }
     }
 }
 
 impl Access for AvroAccess<'_> {
     fn access<'a>(&'a self, path: &[&str], type_expected: &DataType) -> AccessResult<DatumCow<'a>> {
         let mut value = self.value;
-        let mut options: AvroParseOptions<'_> = self.options.clone();
+        let mut unresolved_schema = self.options.root_schema;
 
         debug_assert!(
             path.len() == 1
@@ -401,22 +402,24 @@ impl Access for AvroAccess<'_> {
                     // },
                     // ...]
                     value = v;
-                    let Schema::Union(u) = options.schema else {
+                    let Schema::Union(u) = self.options.inner.lookup_ref(unresolved_schema) else {
                         return Err(create_error());
                     };
                     let Some(schema) = get_nullable_union_inner(u) else {
                         return Err(create_error());
                     };
-                    options.schema = schema;
+                    unresolved_schema = schema;
                     continue;
                 }
                 Value::Record(fields) => {
-                    let Schema::Record(record_schema) = &options.schema else {
+                    let Schema::Record(record_schema) =
+                        self.options.inner.lookup_ref(unresolved_schema)
+                    else {
                         return Err(create_error());
                     };
                     if let Some(idx) = record_schema.lookup.get(key) {
                         value = &fields[*idx].1;
-                        options.schema = &record_schema.fields[*idx].schema;
+                        unresolved_schema = &record_schema.fields[*idx].schema;
                         i += 1;
                         continue;
                     }
@@ -426,7 +429,9 @@ impl Access for AvroAccess<'_> {
             Err(create_error())?;
         }
 
-        options.convert_to_datum(value, type_expected)
+        self.options
+            .inner
+            .convert_to_datum(unresolved_schema, value, type_expected)
     }
 }
 
@@ -941,7 +946,8 @@ mod tests {
         shape: &DataType,
     ) -> anyhow::Result<Datum> {
         Ok(AvroParseOptions::create(value_schema)
-            .convert_to_datum(&value, shape)?
+            .inner
+            .convert_to_datum(value_schema, &value, shape)?
             .to_owned_datum())
     }
 
@@ -982,11 +988,7 @@ mod tests {
         .unwrap();
         let bytes = vec![0x3f, 0x3f, 0x3f, 0x3f, 0x3f, 0x3f, 0x3f];
         let value = Value::Decimal(AvroDecimal::from(bytes));
-        let options = AvroParseOptions::create(&schema);
-        let resp = options
-            .convert_to_datum(&value, &DataType::Decimal)
-            .unwrap()
-            .to_owned_datum();
+        let resp = from_avro_value(value, &schema, &DataType::Decimal).unwrap();
         assert_eq!(
             resp,
             Some(ScalarImpl::Decimal(Decimal::Normalized(
@@ -1022,11 +1024,7 @@ mod tests {
             ("value".to_string(), Value::Bytes(vec![0x01, 0x02, 0x03])),
         ]);
 
-        let options = AvroParseOptions::create(&schema);
-        let resp = options
-            .convert_to_datum(&value, &DataType::Decimal)
-            .unwrap()
-            .to_owned_datum();
+        let resp = from_avro_value(value, &schema, &DataType::Decimal).unwrap();
         assert_eq!(resp, Some(ScalarImpl::Decimal(Decimal::from(66051))));
     }
 }
diff --git a/src/connector/codec/src/decoder/avro/schema.rs b/src/connector/codec/src/decoder/avro/schema.rs
index d226098f63d4..c6a43decc78f 100644
--- a/src/connector/codec/src/decoder/avro/schema.rs
+++ b/src/connector/codec/src/decoder/avro/schema.rs
@@ -35,18 +35,12 @@ use super::get_nullable_union_inner;
 pub struct ResolvedAvroSchema {
     /// Should be used for parsing bytes into Avro value
     pub original_schema: Arc<Schema>,
-    /// Should be used for type mapping from Avro value to RisingWave datum
-    pub resolved_schema: Schema,
 }
 
 impl ResolvedAvroSchema {
     pub fn create(schema: Arc<Schema>) -> AvroResult<Self> {
-        let resolver = ResolvedSchema::try_from(schema.as_ref())?;
-        // todo: to_resolved may cause stackoverflow if there's a loop in the schema
-        let resolved_schema = resolver.to_resolved(schema.as_ref())?;
         Ok(Self {
             original_schema: schema,
-            resolved_schema,
         })
     }
 }
diff --git a/src/connector/src/parser/avro/parser.rs b/src/connector/src/parser/avro/parser.rs
index d4040a8420de..472c416def8b 100644
--- a/src/connector/src/parser/avro/parser.rs
+++ b/src/connector/src/parser/avro/parser.rs
@@ -49,7 +49,7 @@ impl AccessBuilder for AvroAccessBuilder {
         self.value = self.parse_avro_value(&payload).await?;
         Ok(AccessImpl::Avro(AvroAccess::new(
             self.value.as_ref().unwrap(),
-            AvroParseOptions::create(&self.schema.resolved_schema),
+            AvroParseOptions::create(&self.schema.original_schema),
         )))
     }
 }
diff --git a/src/connector/src/parser/debezium/avro_parser.rs b/src/connector/src/parser/debezium/avro_parser.rs
index fd174a9dfe6e..4058f1d331c4 100644
--- a/src/connector/src/parser/debezium/avro_parser.rs
+++ b/src/connector/src/parser/debezium/avro_parser.rs
@@ -57,7 +57,7 @@ impl AccessBuilder for DebeziumAvroAccessBuilder {
             // Assumption: Key will not contain reference, so unresolved schema can work here.
             AvroParseOptions::create(match self.encoding_type {
                 EncodingType::Key => self.key_schema.as_mut().unwrap(),
-                EncodingType::Value => &self.schema.resolved_schema,
+                EncodingType::Value => &self.schema.original_schema,
             }),
         )))
     }


Test patch:
diff --git a/e2e_test/source_inline/kafka/avro/ref.slt b/e2e_test/source_inline/kafka/avro/ref.slt
index f9a09a0e8899..e3659953f677 100644
--- a/e2e_test/source_inline/kafka/avro/ref.slt
+++ b/e2e_test/source_inline/kafka/avro/ref.slt
@@ -109,11 +109,102 @@ select
   (bar).b.y
 from s;
 ----
-3 4 5 6 NULL NULL NULL NULL
+3 4 5 6 6 5 4 3
 
-# Parsing of column `bar` fails even with ints because now `schema` is required.
-# This will be fully supported in the next PR
-# 3 4 5 6 6 5 4 3
+
+statement ok
+drop source s;
+
+
+system ok
+curl -X DELETE "${RISEDEV_SCHEMA_REGISTRY_URL}/subjects/avro-ref-value"
+
+
+system ok
+curl -X DELETE "${RISEDEV_SCHEMA_REGISTRY_URL}/subjects/avro-ref-value?permanent=true"
+
+
+system ok
+rpk topic delete 'avro-ref'
+
+
+system ok
+rpk topic create avro-ref
+
+
+system ok
+sr_register avro-ref-value AVRO <<EOF
+{
+  "type": "record",
+  "name": "Root",
+  "fields": [
+    {
+      "name": "foo",
+      "type": {
+        "type": "record",
+        "name": "Seg",
+        "fields": [
+          {
+            "name": "a",
+            "type": {
+              "type": "record",
+              "name": "Point",
+              "fields": [
+                {
+                  "name": "x",
+                  "type": {
+                    "type": "bytes",
+                    "logicalType": "decimal",
+                    "precision": 4,
+                    "scale": 2
+                  }
+                },
+                {
+                  "name": "y",
+                  "type": "int"
+                }
+              ]
+            }
+          },
+          {
+            "name": "b",
+            "type": "Point"
+          }
+        ]
+      }
+    },
+    {
+      "name": "bar",
+      "type": "Seg"
+    }
+  ]
+}
+EOF
+
+
+statement ok
+create source s WITH (${RISEDEV_KAFKA_WITH_OPTIONS_COMMON}, topic = 'avro-ref') FORMAT PLAIN ENCODE AVRO (schema.registry = '${RISEDEV_SCHEMA_REGISTRY_URL}');
+
+
+system ok
+rpk topic produce avro-ref --schema-id=topic <<EOF
+{"foo":{"a":{"x":"\u0001\u002c","y":4},"b":{"x":"\u0001\u00f4","y":6}},"bar":{"a":{"x":"\u0002\u0058","y":5},"b":{"x":"\u0001\u0090","y":3}}}
+EOF
+
+
+query RIRIRIRI
+select
+  (foo).a.x,
+  (foo).a.y,
+  (foo).b.x,
+  (foo).b.y,
+  (bar).a.x,
+  (bar).a.y,
+  (bar).b.x,
+  (bar).b.y
+from s;
+----
+3.00 4 5.00 6 6.00 5 4.00 3
 
 
 statement ok
diff --git a/src/connector/codec/tests/integration_tests/avro.rs b/src/connector/codec/tests/integration_tests/avro.rs
index 2221917cc2b0..c0a0b1d49e2c 100644
--- a/src/connector/codec/tests/integration_tests/avro.rs
+++ b/src/connector/codec/tests/integration_tests/avro.rs
@@ -94,7 +94,7 @@ fn check(
     // manually implement some logic in AvroAccessBuilder, and some in PlainParser::parse_inner
     let mut data_str = vec![];
     for data in avro_data {
-        let parser = AvroParseOptions::create(&resolved_schema.resolved_schema);
+        let parser = AvroParseOptions::create(&resolved_schema.original_schema);
 
         match config.data_encoding {
             TestDataEncoding::Json => todo!(),