Repository: substrait-io/substrait-java
Pull request #338: fix(isthmus): concat of different string types
Author: login: bestbeforetoday
Created: 2025-03-05T14:36:46Z | Merged: 2025-04-23T14:56:00Z

The type matching used when mapping Calcite to Substrait expressions required parameters and return type of the concat function to be either all varchar or all string. Any mixing of types caused a failure.

This change relaxes the type matching to allow string types (varchar, char and string literal) to be used interchangeably.

Closes #69
Contributes to #105

Linked issue #69: [ISTHMUS] [TPCDS] Fix Concat scalar function support
For example:
- `q5` has the expression: `'web_site' || web_site_id as id`
- `q66` has the expression: `'DHL' || ',' || 'BARIAN' as ship_carriers`
- `q80` has expression using ||

Implicit casting is not 

Queries in tpcds having this issue: `[5, 66, 80, 84]`

How to reproduce:
 Add a test to `TpcdsQueryNoValidation` like this:

```java
@ParameterizedTest
  @ValueSource(ints = {5})
  public void tpcdsQ5(int query) throws Exception {
    testQuery(query);
  }
```

On fixing, please move the fixed queries from the `tpcdsFailure` list to the `tpcdsSuccess` list

Diff:
diff --git a/isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java b/isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java
index 123c9bd92..8817ddca2 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java
@@ -39,6 +39,11 @@ public int getMaxNumericPrecision() {
     return 38;
   }
 
+  @Override
+  public boolean shouldConvertRaggedUnionTypesToVarying() {
+    return true;
+  }
+
   public static RelDataTypeFactory createTypeFactory() {
     return new JavaTypeFactoryImpl(TYPE_SYSTEM);
   }
diff --git a/isthmus/src/main/java/io/substrait/isthmus/expression/AggregateFunctionConverter.java b/isthmus/src/main/java/io/substrait/isthmus/expression/AggregateFunctionConverter.java
index a62f7c0e7..29039be8d 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/expression/AggregateFunctionConverter.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/expression/AggregateFunctionConverter.java
@@ -52,7 +52,7 @@ public AggregateFunctionConverter(
   protected AggregateFunctionInvocation generateBinding(
       WrappedAggregateCall call,
       SimpleExtension.AggregateFunctionVariant function,
-      List<FunctionArg> arguments,
+      List<? extends FunctionArg> arguments,
       Type outputType) {
     AggregateCall agg = call.getUnderlying();
 
diff --git a/isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java b/isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
index 9e0e496b0..a39c844f1 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
@@ -25,6 +25,7 @@
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
 import java.util.function.Function;
@@ -149,7 +150,6 @@ protected class FunctionFinder {
     private final SqlOperator operator;
     private final List<F> functions;
     private final Map<String, F> directMap;
-    private final SignatureMatcher<F> matcher;
     private final Optional<SingularArgumentMatcher<F>> singularInputType;
     private final Util.IntRange argRange;
 
@@ -161,7 +161,6 @@ public FunctionFinder(String name, SqlOperator operator, List<F> functions) {
           Util.IntRange.of(
               functions.stream().mapToInt(t -> t.getRange().getStartInclusive()).min().getAsInt(),
               functions.stream().mapToInt(t -> t.getRange().getEndExclusive()).max().getAsInt());
-      this.matcher = getSignatureMatcher(operator, functions);
       this.singularInputType = getSingularInputType(functions);
       var directMap = ImmutableMap.<String, F>builder();
       for (var func : functions) {
@@ -178,21 +177,18 @@ public boolean allowedArgCount(int count) {
       return argRange.within(count);
     }
 
-    private static <F extends SimpleExtension.Function> SignatureMatcher<F> getSignatureMatcher(
-        SqlOperator operator, List<F> functions) {
-      return (inputTypes, outputType) -> {
-        for (F function : functions) {
-          List<SimpleExtension.Argument> args = function.requiredArguments();
-          // Make sure that arguments & return are within bounds and match the types
-          if (function.returnType() instanceof ParameterizedType
-              && isMatch(outputType, (ParameterizedType) function.returnType())
-              && inputTypesSatisfyDefinedArguments(inputTypes, args)) {
-            return Optional.of(function);
-          }
+    private Optional<F> signatureMatch(List<Type> inputTypes, Type outputType) {
+      for (F function : functions) {
+        List<SimpleExtension.Argument> args = function.requiredArguments();
+        // Make sure that arguments & return are within bounds and match the types
+        if (function.returnType() instanceof ParameterizedType
+            && isMatch(outputType, (ParameterizedType) function.returnType())
+            && inputTypesMatchDefinedArguments(inputTypes, args)) {
+          return Optional.of(function);
         }
+      }
 
-        return Optional.empty();
-      };
+      return Optional.empty();
     }
 
     /**
@@ -208,7 +204,7 @@ && inputTypesSatisfyDefinedArguments(inputTypes, args)) {
      * @param args expected arguments as defined in a {@link SimpleExtension.Function}
      * @return true if the {@code inputTypes} satisfy the {@code args}, false otherwise
      */
-    private static boolean inputTypesSatisfyDefinedArguments(
+    private static boolean inputTypesMatchDefinedArguments(
         List<Type> inputTypes, List<SimpleExtension.Argument> args) {
 
       Map<String, Set<Type>> wildcardToType = new HashMap<>();
@@ -318,7 +314,7 @@ private Stream<String> matchKeys(List<RexNode> rexOperands, List<String> opTypes
 
       assert (rexOperands.size() == opTypes.size());
 
-      if (rexOperands.size() == 0) {
+      if (rexOperands.isEmpty()) {
         return Stream.of("");
       } else {
         List<List<String>> argTypeLists =
@@ -357,13 +353,12 @@ public Optional<T> attemptMatch(C call, Function<RexNode, Expression> topLevelCo
       // try to do a direct match
       List<String> typeStrings =
           opTypes.stream().map(t -> t.accept(ToTypeString.INSTANCE)).collect(Collectors.toList());
-      Stream<String> possibleKeys =
-          matchKeys(call.getOperands().collect(Collectors.toList()), typeStrings);
+      Stream<String> possibleKeys = matchKeys(operandsList, typeStrings);
 
       Optional<String> directMatchKey =
           possibleKeys
               .map(argList -> name + ":" + argList)
-              .filter(k -> directMap.containsKey(k))
+              .filter(directMap::containsKey)
               .findFirst();
 
       if (directMatchKey.isPresent()) {
@@ -376,14 +371,13 @@ public Optional<T> attemptMatch(C call, Function<RexNode, Expression> topLevelCo
                       RexNode r = operandsList.get(i);
                       Expression o = operands.get(i);
                       if (EnumConverter.isEnumValue(r)) {
-                        return EnumConverter.fromRex(variant, (RexLiteral) r, i)
-                            .orElseGet(() -> null);
+                        return EnumConverter.fromRex(variant, (RexLiteral) r, i).orElse(null);
                       } else {
                         return o;
                       }
                     })
                 .collect(Collectors.toList());
-        boolean allArgsMapped = funcArgs.stream().filter(e -> e == null).findFirst().isEmpty();
+        boolean allArgsMapped = funcArgs.stream().filter(Objects::isNull).findFirst().isEmpty();
         if (allArgsMapped) {
           return Optional.of(generateBinding(call, variant, funcArgs, outputType));
         } else {
@@ -413,53 +407,35 @@ private Optional<T> matchByLeastRestrictive(
         return Optional.empty();
       }
       Type type = typeConverter.toSubstrait(leastRestrictive);
-      var out = singularInputType.get().tryMatch(type, outputType);
-
-      if (out.isPresent()) {
-        var declaration = out.get();
-        var coercedArgs = coerceArguments(operands, type);
-        declaration.validateOutputType(coercedArgs, outputType);
-        return Optional.of(
-            generateBinding(
-                call,
-                out.get(),
-                coercedArgs.stream().map(FunctionArg.class::cast).collect(Collectors.toList()),
-                outputType));
-      }
-      return Optional.empty();
+      var out = singularInputType.orElseThrow().tryMatch(type, outputType);
+
+      return out.map(
+          declaration -> {
+            var coercedArgs = coerceArguments(operands, type);
+            declaration.validateOutputType(coercedArgs, outputType);
+            return generateBinding(call, out.get(), coercedArgs, outputType);
+          });
     }
 
-    private Optional<T> matchCoerced(C call, Type outputType, List<Expression> operands) {
-
+    private Optional<T> matchCoerced(C call, Type outputType, List<Expression> expressions) {
       // Convert the operands to the proper Substrait type
-      List<Type> allTypes =
+      List<Type> operandTypes =
           call.getOperands()
               .map(RexNode::getType)
               .map(typeConverter::toSubstrait)
               .collect(Collectors.toList());
 
-      // See if all the input types match the function
-      Optional<F> matchFunction = this.matcher.tryMatch(allTypes, outputType);
-      if (matchFunction.isPresent()) {
-        List<Expression> coerced =
-            Streams.zip(
-                    operands.stream(),
-                    call.getOperands(),
-                    (a, b) -> {
-                      Type type = typeConverter.toSubstrait(b.getType());
-                      return coerceArgument(a, type);
-                    })
-                .collect(Collectors.toList());
-
-        return Optional.of(
-            generateBinding(
-                call,
-                matchFunction.get(),
-                coerced.stream().map(FunctionArg.class::cast).collect(Collectors.toList()),
-                outputType));
+      // See if all the input types can be made to match the function
+      Optional<F> matchFunction = signatureMatch(operandTypes, outputType);
+      if (matchFunction.isEmpty()) {
+        return Optional.empty();
       }
 
-      return Optional.empty();
+      var coercedArgs =
+          Streams.zip(
+                  expressions.stream(), operandTypes.stream(), FunctionConverter::coerceArgument)
+              .collect(Collectors.toList());
+      return Optional.of(generateBinding(call, matchFunction.get(), coercedArgs, outputType));
     }
 
     protected String getName() {
@@ -481,56 +457,30 @@ public interface GenericCall {
    * Coerced types according to an expected output type. Coercion is only done for type mismatches,
    * not for nullability or parameter mismatches.
    */
-  private static List<Expression> coerceArguments(List<Expression> arguments, Type type) {
-    return arguments.stream().map(a -> coerceArgument(a, type)).collect(Collectors.toList());
+  private static List<Expression> coerceArguments(List<Expression> arguments, Type targetType) {
+    return arguments.stream().map(a -> coerceArgument(a, targetType)).collect(Collectors.toList());
   }
 
   private static Expression coerceArgument(Expression argument, Type type) {
-    var typeMatches = isMatch(type, argument.getType());
-    if (!typeMatches) {
-      return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
+    if (isMatch(type, argument.getType())) {
+      return argument;
     }
-    return argument;
+
+    return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
   }
 
   protected abstract T generateBinding(
-      C call, F function, List<FunctionArg> arguments, Type outputType);
+      C call, F function, List<? extends FunctionArg> arguments, Type outputType);
 
-  public interface SingularArgumentMatcher<F> {
+  @FunctionalInterface
+  private interface SingularArgumentMatcher<F> {
     Optional<F> tryMatch(Type type, Type outputType);
   }
 
-  public interface SignatureMatcher<F> {
-    Optional<F> tryMatch(List<Type> types, Type outputType);
-  }
-
-  private static SignatureMatcher chainedSignature(SignatureMatcher... matchers) {
-    return switch (matchers.length) {
-      case 0 -> (types, outputType) -> Optional.empty();
-      case 1 -> matchers[0];
-      default -> (types, outputType) -> {
-        for (SignatureMatcher m : matchers) {
-          var t = m.tryMatch(types, outputType);
-          if (t.isPresent()) {
-            return t;
-          }
-        }
-        return Optional.empty();
-      };
-    };
-  }
-
-  private static boolean isMatch(Type inputType, ParameterizedType type) {
-    if (type.isWildcard()) {
-      return true;
-    }
-    return inputType.accept(new IgnoreNullableAndParameters(type));
-  }
-
-  private static boolean isMatch(ParameterizedType inputType, ParameterizedType type) {
-    if (type.isWildcard()) {
+  private static boolean isMatch(ParameterizedType actualType, ParameterizedType targetType) {
+    if (targetType.isWildcard()) {
       return true;
     }
-    return inputType.accept(new IgnoreNullableAndParameters(type));
+    return actualType.accept(new IgnoreNullableAndParameters(targetType));
   }
 }
diff --git a/isthmus/src/main/java/io/substrait/isthmus/expression/ScalarFunctionConverter.java b/isthmus/src/main/java/io/substrait/isthmus/expression/ScalarFunctionConverter.java
index 798e66f10..c04d3aa51 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/expression/ScalarFunctionConverter.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/expression/ScalarFunctionConverter.java
@@ -60,7 +60,7 @@ public Optional<Expression> convert(
   protected Expression generateBinding(
       WrappedScalarCall call,
       SimpleExtension.ScalarFunctionVariant function,
-      List<FunctionArg> arguments,
+      List<? extends FunctionArg> arguments,
       Type outputType) {
     return Expression.ScalarFunctionInvocation.builder()
         .outputType(outputType)
diff --git a/isthmus/src/main/java/io/substrait/isthmus/expression/WindowFunctionConverter.java b/isthmus/src/main/java/io/substrait/isthmus/expression/WindowFunctionConverter.java
index 2d3b42051..30b013399 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/expression/WindowFunctionConverter.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/expression/WindowFunctionConverter.java
@@ -52,7 +52,7 @@ public WindowFunctionConverter(
   protected Expression.WindowFunctionInvocation generateBinding(
       WrappedWindowCall call,
       SimpleExtension.WindowFunctionVariant function,
-      List<FunctionArg> arguments,
+      List<? extends FunctionArg> arguments,
       Type outputType) {
     RexOver over = call.over;
     RexWindow window = over.getWindow();
diff --git a/isthmus/src/main/java/io/substrait/isthmus/expression/WindowRelFunctionConverter.java b/isthmus/src/main/java/io/substrait/isthmus/expression/WindowRelFunctionConverter.java
index cd254d551..a1e9bff33 100644
--- a/isthmus/src/main/java/io/substrait/isthmus/expression/WindowRelFunctionConverter.java
+++ b/isthmus/src/main/java/io/substrait/isthmus/expression/WindowRelFunctionConverter.java
@@ -51,7 +51,7 @@ public WindowRelFunctionConverter(
   protected ConsistentPartitionWindow.WindowRelFunctionInvocation generateBinding(
       WrappedWindowRelCall call,
       SimpleExtension.WindowFunctionVariant function,
-      List<FunctionArg> arguments,
+      List<? extends FunctionArg> arguments,
       Type outputType) {
     Window.RexWinAggCall over = call.getWinAggCall();
 
diff --git a/isthmus/src/test/java/io/substrait/isthmus/FunctionConversionTest.java b/isthmus/src/test/java/io/substrait/isthmus/FunctionConversionTest.java
index 4c06d91b1..51c8e7bdc 100644
--- a/isthmus/src/test/java/io/substrait/isthmus/FunctionConversionTest.java
+++ b/isthmus/src/test/java/io/substrait/isthmus/FunctionConversionTest.java
@@ -258,4 +258,19 @@ public void unsupportedExtractDateWithIndexing() {
     assertThrows(
         UnsupportedOperationException.class, () -> reqReqDateFn.accept(expressionRexConverter));
   }
+
+  @Test
+  public void concatStringLiteralAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select 'part_'||P_NAME from PART");
+  }
+
+  @Test
+  public void concatCharAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select P_BRAND||P_NAME from PART");
+  }
+
+  @Test
+  public void concatStringLiteralAndChar() throws Exception {
+    assertProtoPlanRoundrip("select 'brand_'||P_BRAND from PART");
+  }
 }
diff --git a/isthmus/src/test/java/io/substrait/isthmus/TpcdsQueryNoValidation.java b/isthmus/src/test/java/io/substrait/isthmus/TpcdsQueryNoValidation.java
index 9549951e2..60086e150 100644
--- a/isthmus/src/test/java/io/substrait/isthmus/TpcdsQueryNoValidation.java
+++ b/isthmus/src/test/java/io/substrait/isthmus/TpcdsQueryNoValidation.java
@@ -1,9 +1,11 @@
 package io.substrait.isthmus;
 
 import com.google.protobuf.util.JsonFormat;
+import java.util.Set;
+import java.util.stream.IntStream;
 import org.apache.calcite.adapter.tpcds.TpcdsSchema;
 import org.junit.jupiter.params.ParameterizedTest;
-import org.junit.jupiter.params.provider.ValueSource;
+import org.junit.jupiter.params.provider.MethodSource;
 
 /**
  *
@@ -27,33 +29,24 @@
  */
 public class TpcdsQueryNoValidation extends PlanTestBase {
 
+  static final Set<Integer> EXCLUDED =
+      Set.of(2, 9, 12, 20, 27, 36, 47, 51, 53, 57, 63, 70, 86, 89, 98);
+
+  static IntStream testCases() {
+    return IntStream.rangeClosed(1, 99).filter(n -> !EXCLUDED.contains(n));
+  }
+
   /**
    * This test only validates that generating substrait plans for TPC-DS queries does not fail. As
    * of now this test does not validate correctness of the generated plan
    */
-  private void testQuery(int i) throws Exception {
+  @ParameterizedTest
+  @MethodSource("testCases")
+  void testQuery(int i) throws Exception {
     SqlToSubstrait s = new SqlToSubstrait();
     TpcdsSchema schema = new TpcdsSchema(1.0);
     String sql = asString(String.format("tpcds/queries/%02d.sql", i));
     var plan = s.execute(sql, "tpcds", schema);
     System.out.println(JsonFormat.printer().print(plan));
   }
-
-  @ParameterizedTest
-  @ValueSource(
-      ints = {
-        1, 3, 4, 6, 7, 8, 10, 11, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 25, 26, 28, 29, 30,
-        31, 32, 33, 34, 35, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 48, 49, 50, 52, 54, 55, 56, 58,
-        59, 60, 61, 62, 64, 65, 67, 68, 69, 71, 72, 73, 74, 75, 76, 77, 78, 79, 81, 82, 83, 85, 87,
-        88, 90, 92, 93, 94, 95, 96, 97, 99
-      })
-  public void tpcdsSuccess(int query) throws Exception {
-    testQuery(query);
-  }
-
-  @ParameterizedTest
-  @ValueSource(ints = {2, 5, 9, 12, 20, 27, 36, 47, 51, 53, 57, 63, 66, 70, 80, 84, 86, 89, 91, 98})
-  public void tpcdsFailure(int query) throws Exception {
-    // testQuery(query);
-  }
 }


Reviews:
user: login: vbarua
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T04:44:55Z
user: login: bestbeforetoday
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T10:31:43Z
user: login: vbarua
body: I just noticed that this was ready for re-review. Took a preliminary look at everything except the FunctionConverter, for which I need fresh eyes.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T23:17:06Z
user: login: vbarua
body: Overall this looks reasonable to me.

@bestbeforetoday could you confirm for me that there are no functionality changes in `FunctionConverter.java`? From what I reviewed, it looks like there are only cleanups and code simplifications.

Additionally, could you move the tests in ExpressionConvertabilityTest.java as per https://github.com/substrait-io/substrait-java/pull/338/files#r2038494522 and fix the conflicts.

Once those things are done, I'm happy to merge this.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T23:10:24Z
user: login: vbarua
body: Marking as waiting for changes.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-17T15:26:40Z
user: login: bestbeforetoday
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T12:08:36Z
user: login: bestbeforetoday
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T13:01:43Z
user: login: vbarua
body: Changes look good, I will merge this later today.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-22T16:54:36Z

Review comments:
diff_hunk: @@ -65,6 +65,21 @@ public void inPredicate() throws IOException, SqlParseException {
             + "(SELECT L_SUPPKEY from LINEITEM where L_SUPPKEY < L_ORDERKEY)");
   }
 
+  @Test
+  public void concatStringLiteralAndVarchar() throws IOException, SqlParseException {
+    assertFullRoundTrip("select 'part_'||P_NAME from PART");
+  }
path: isthmus/src/test/java/io/substrait/isthmus/ExpressionConvertabilityTest.java
user: login: vbarua
body: When I look at this test I'm actually somewhat suprised that this doesn't throw an exception of some kind in the Substrait layer.

The POJO this produces has the following scalar function invocation
```
ScalarFunctionInvocation{
    declaration=concat:vchar,
    arguments=[
        FixedCharLiteral{nullable=false, value=part_},
        FieldReference{
            segments=[StructField{offset=1}],
            type=VarChar{nullable=true, length=55}}
    ]
    , options=[],
    outputType=VarChar{nullable=true, length=60}
}
```
As I understand it, this is an invalid invocation of `concat:vchar` from [functions_string.yaml](https://github.com/substrait-io/substrait/blob/10d53ca376cde9342a74d3480fcd2e77109d65cd/extensions/functions_string.yaml#L12-L21) 
because all arguments _must_ be varchars. Substrait, intentionally, doesn't have any form of implicit casting to allow calls to this function that mix string types. 

While SQL may allow us to `concat` different string types together, if we want to encode this into valid Substrait we'll need to upcast the arguments to the same type when translating into Substrait.
created_at: 2025-03-06T04:44:55Z
author_association: MEMBER
diff_hunk: @@ -65,6 +65,21 @@ public void inPredicate() throws IOException, SqlParseException {
             + "(SELECT L_SUPPKEY from LINEITEM where L_SUPPKEY < L_ORDERKEY)");
   }
 
+  @Test
+  public void concatStringLiteralAndVarchar() throws IOException, SqlParseException {
+    assertFullRoundTrip("select 'part_'||P_NAME from PART");
+  }
path: isthmus/src/test/java/io/substrait/isthmus/ExpressionConvertabilityTest.java
user: login: bestbeforetoday
body: Thank you for the feedback. The code changed is used by function matching code for cases where the types don't exactly match. Since the change didn't break any tests I (wrongly!) assumed it was successfully fixing the problem. I will dig more into the matching to find out if any type coercion should be happening and, depending on the result, either fix it or implement it.
created_at: 2025-03-06T10:31:43Z
author_association: MEMBER
diff_hunk: @@ -39,6 +39,11 @@ public int getMaxNumericPrecision() {
     return 38;
   }
 
+  @Override
+  public boolean shouldConvertRaggedUnionTypesToVarying() {
+    return true;
+  }
path: isthmus/src/main/java/io/substrait/isthmus/SubstraitTypeSystem.java
user: login: vbarua
body: Did this turn out to do what we needed in terms of upcasting the types in Calcite?
created_at: 2025-04-10T22:30:29Z
author_association: MEMBER
diff_hunk: @@ -65,6 +65,21 @@ public void inPredicate() throws IOException, SqlParseException {
             + "(SELECT L_SUPPKEY from LINEITEM where L_SUPPKEY < L_ORDERKEY)");
   }
 
+  @Test
+  public void concatStringLiteralAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select 'part_'||P_NAME from PART");
+  }
+
+  @Test
+  public void concatCharAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select P_BRAND||P_NAME from PART");
+  }
+
+  @Test
+  public void concatStringLiteralAndChar() throws Exception {
+    assertProtoPlanRoundrip("select 'brand_'||P_BRAND from PART");
+  }
path: isthmus/src/test/java/io/substrait/isthmus/ExpressionConvertabilityTest.java
user: login: vbarua
body: minor: Could you move these tests to the (new) [FunctionConversionTest.java](https://github.com/substrait-io/substrait-java/blob/main/isthmus/src/test/java/io/substrait/isthmus/FunctionConversionTest.java) class.

I'm trying to group tests related to specific functions there. 
created_at: 2025-04-10T22:51:48Z
author_association: MEMBER
diff_hunk: @@ -65,6 +65,21 @@ public void inPredicate() throws IOException, SqlParseException {
             + "(SELECT L_SUPPKEY from LINEITEM where L_SUPPKEY < L_ORDERKEY)");
   }
 
+  @Test
+  public void concatStringLiteralAndVarchar() throws IOException, SqlParseException {
+    assertFullRoundTrip("select 'part_'||P_NAME from PART");
+  }
path: isthmus/src/test/java/io/substrait/isthmus/ExpressionConvertabilityTest.java
user: login: vbarua
body: The type system change you added definitely helped.
```
ScalarFunctionInvocation{
    declaration=concat:vchar,
    arguments=[
        Cast{type=VarChar{nullable=true, length=55}, input=FixedCharLiteral{nullable=false, value=part_}, failureBehavior=THROW_EXCEPTION},
        FieldReference{segments=[StructField{offset=1}], type=VarChar{nullable=true, length=55}}
    ],
    options=[],
    outputType=VarChar{nullable=true, length=60}
}
```

Both inputs are now varchars 🎉 

The output schema doesn't quite match Substrait still, which would expect `VarChar<55>` instead of `VarChar<60>`.

That's happening because the Calcite definition of [CONCAT](https://github.com/apache/calcite/blob/61e6dd11706aed0ee617a251643022b331c7d466/core/src/main/java/org/apache/calcite/sql/fun/SqlStdOperatorTable.java#L260-L281)
uses [ReturnTypes.DYADIC_STRING_SUM_PRECISION_NULLABLE](https://github.com/apache/calcite/blob/61e6dd11706aed0ee617a251643022b331c7d466/core/src/main/java/org/apache/calcite/sql/type/ReturnTypes.java#L1342-L1347) for its return type inference which ends up doing the following
> Type-inference strategy for concatenating two string arguments. The result type of a call is:
> * the same type as the input types but with the combined length of the two first types
> * if types are of char type the type with the highest coercibility will be used
> * result is varying if either input is; otherwise fixed
> 
> Pre-requisites:
> * input types must be of the same string type
> * types must be comparable without casting

I think we can tackle that as part of https://github.com/substrait-io/substrait-java/issues/379 


created_at: 2025-04-10T23:05:07Z
author_association: MEMBER
diff_hunk: @@ -177,21 +176,18 @@ public boolean allowedArgCount(int count) {
       return argRange.within(count);
     }
 
-    private static <F extends SimpleExtension.Function> SignatureMatcher<F> getSignatureMatcher(
-        SqlOperator operator, List<F> functions) {
-      return (inputTypes, outputType) -> {
-        for (F function : functions) {
-          List<SimpleExtension.Argument> args = function.requiredArguments();
-          // Make sure that arguments & return are within bounds and match the types
-          if (function.returnType() instanceof ParameterizedType
-              && isMatch(outputType, (ParameterizedType) function.returnType())
-              && inputTypesSatisfyDefinedArguments(inputTypes, args)) {
-            return Optional.of(function);
-          }
+    private Optional<F> signatureMatch(List<Type> inputTypes, Type outputType) {
+      for (F function : functions) {
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: REVIEW NOTES: This works because `operator` was never used, and `getSignatureMathcher` was only ever called with the `functions` attached the `FunctionFinder` class.
created_at: 2025-04-16T22:37:24Z
author_association: MEMBER
diff_hunk: @@ -411,53 +404,35 @@ private Optional<T> matchByLeastRestrictive(
         return Optional.empty();
       }
       Type type = typeConverter.toSubstrait(leastRestrictive);
-      var out = singularInputType.get().tryMatch(type, outputType);
-
-      if (out.isPresent()) {
-        var declaration = out.get();
-        var coercedArgs = coerceArguments(operands, type);
-        declaration.validateOutputType(coercedArgs, outputType);
-        return Optional.of(
-            generateBinding(
-                call,
-                out.get(),
-                coercedArgs.stream().map(FunctionArg.class::cast).collect(Collectors.toList()),
-                outputType));
-      }
-      return Optional.empty();
+      var out = singularInputType.orElseThrow().tryMatch(type, outputType);
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: REVIEW NOTE: `get()` has the same behaviour as `orElseThrow()`
created_at: 2025-04-16T22:43:50Z
author_association: MEMBER
diff_hunk: @@ -411,53 +404,35 @@ private Optional<T> matchByLeastRestrictive(
         return Optional.empty();
       }
       Type type = typeConverter.toSubstrait(leastRestrictive);
-      var out = singularInputType.get().tryMatch(type, outputType);
-
-      if (out.isPresent()) {
-        var declaration = out.get();
-        var coercedArgs = coerceArguments(operands, type);
-        declaration.validateOutputType(coercedArgs, outputType);
-        return Optional.of(
-            generateBinding(
-                call,
-                out.get(),
-                coercedArgs.stream().map(FunctionArg.class::cast).collect(Collectors.toList()),
-                outputType));
-      }
-      return Optional.empty();
+      var out = singularInputType.orElseThrow().tryMatch(type, outputType);
+
+      return out.map(
+          declaration -> {
+            var coercedArgs = coerceArguments(operands, type);
+            declaration.validateOutputType(coercedArgs, outputType);
+            return generateBinding(call, out.get(), coercedArgs, outputType);
+          });
     }
 
-    private Optional<T> matchCoerced(C call, Type outputType, List<Expression> operands) {
-
+    private Optional<T> matchCoerced(C call, Type outputType, List<Expression> expressions) {
       // Convert the operands to the proper Substrait type
-      List<Type> allTypes =
+      List<Type> operandTypes =
           call.getOperands()
               .map(RexNode::getType)
               .map(typeConverter::toSubstrait)
               .collect(Collectors.toList());
 
-      // See if all the input types match the function
-      Optional<F> matchFunction = this.matcher.tryMatch(allTypes, outputType);
-      if (matchFunction.isPresent()) {
-        List<Expression> coerced =
-            Streams.zip(
-                    operands.stream(),
-                    call.getOperands(),
-                    (a, b) -> {
-                      Type type = typeConverter.toSubstrait(b.getType());
-                      return coerceArgument(a, type);
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: 🧹 Nice cleanup re-using `operandTypes` here
created_at: 2025-04-16T22:53:32Z
author_association: MEMBER
diff_hunk: @@ -479,56 +454,30 @@ public interface GenericCall {
    * Coerced types according to an expected output type. Coercion is only done for type mismatches,
    * not for nullability or parameter mismatches.
    */
-  private static List<Expression> coerceArguments(List<Expression> arguments, Type type) {
-    return arguments.stream().map(a -> coerceArgument(a, type)).collect(Collectors.toList());
+  private static List<Expression> coerceArguments(List<Expression> arguments, Type targetType) {
+    return arguments.stream().map(a -> coerceArgument(a, targetType)).collect(Collectors.toList());
   }
 
   private static Expression coerceArgument(Expression argument, Type type) {
-    var typeMatches = isMatch(type, argument.getType());
-    if (!typeMatches) {
-      return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
+    if (isMatch(type, argument.getType())) {
+      return argument;
     }
-    return argument;
+
+    return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
   }
 
   protected abstract T generateBinding(
-      C call, F function, List<FunctionArg> arguments, Type outputType);
+      C call, F function, List<? extends FunctionArg> arguments, Type outputType);
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: 🧹 Good call making `arguments` allow any FunctionArg. Make some of the above code nicer.
created_at: 2025-04-16T22:57:23Z
author_association: MEMBER
diff_hunk: @@ -479,56 +454,30 @@ public interface GenericCall {
    * Coerced types according to an expected output type. Coercion is only done for type mismatches,
    * not for nullability or parameter mismatches.
    */
-  private static List<Expression> coerceArguments(List<Expression> arguments, Type type) {
-    return arguments.stream().map(a -> coerceArgument(a, type)).collect(Collectors.toList());
+  private static List<Expression> coerceArguments(List<Expression> arguments, Type targetType) {
+    return arguments.stream().map(a -> coerceArgument(a, targetType)).collect(Collectors.toList());
   }
 
   private static Expression coerceArgument(Expression argument, Type type) {
-    var typeMatches = isMatch(type, argument.getType());
-    if (!typeMatches) {
-      return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
+    if (isMatch(type, argument.getType())) {
+      return argument;
     }
-    return argument;
+
+    return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
   }
 
   protected abstract T generateBinding(
-      C call, F function, List<FunctionArg> arguments, Type outputType);
+      C call, F function, List<? extends FunctionArg> arguments, Type outputType);
 
-  public interface SingularArgumentMatcher<F> {
+  @FunctionalInterface
+  private interface SingularArgumentMatcher<F> {
     Optional<F> tryMatch(Type type, Type outputType);
   }
 
-  public interface SignatureMatcher<F> {
-    Optional<F> tryMatch(List<Type> types, Type outputType);
-  }
-
-  private static SignatureMatcher chainedSignature(SignatureMatcher... matchers) {
-    return switch (matchers.length) {
-      case 0 -> (types, outputType) -> Optional.empty();
-      case 1 -> matchers[0];
-      default -> (types, outputType) -> {
-        for (SignatureMatcher m : matchers) {
-          var t = m.tryMatch(types, outputType);
-          if (t.isPresent()) {
-            return t;
-          }
-        }
-        return Optional.empty();
-      };
-    };
-  }
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: REVIEW NOTE: this code is unused
created_at: 2025-04-16T23:01:43Z
author_association: MEMBER
diff_hunk: @@ -479,56 +454,30 @@ public interface GenericCall {
    * Coerced types according to an expected output type. Coercion is only done for type mismatches,
    * not for nullability or parameter mismatches.
    */
-  private static List<Expression> coerceArguments(List<Expression> arguments, Type type) {
-    return arguments.stream().map(a -> coerceArgument(a, type)).collect(Collectors.toList());
+  private static List<Expression> coerceArguments(List<Expression> arguments, Type targetType) {
+    return arguments.stream().map(a -> coerceArgument(a, targetType)).collect(Collectors.toList());
   }
 
   private static Expression coerceArgument(Expression argument, Type type) {
-    var typeMatches = isMatch(type, argument.getType());
-    if (!typeMatches) {
-      return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
+    if (isMatch(type, argument.getType())) {
+      return argument;
     }
-    return argument;
+
+    return ExpressionCreator.cast(type, argument, Expression.FailureBehavior.THROW_EXCEPTION);
   }
 
   protected abstract T generateBinding(
-      C call, F function, List<FunctionArg> arguments, Type outputType);
+      C call, F function, List<? extends FunctionArg> arguments, Type outputType);
 
-  public interface SingularArgumentMatcher<F> {
+  @FunctionalInterface
+  private interface SingularArgumentMatcher<F> {
     Optional<F> tryMatch(Type type, Type outputType);
   }
 
-  public interface SignatureMatcher<F> {
-    Optional<F> tryMatch(List<Type> types, Type outputType);
-  }
-
-  private static SignatureMatcher chainedSignature(SignatureMatcher... matchers) {
-    return switch (matchers.length) {
-      case 0 -> (types, outputType) -> Optional.empty();
-      case 1 -> matchers[0];
-      default -> (types, outputType) -> {
-        for (SignatureMatcher m : matchers) {
-          var t = m.tryMatch(types, outputType);
-          if (t.isPresent()) {
-            return t;
-          }
-        }
-        return Optional.empty();
-      };
-    };
-  }
-
-  private static boolean isMatch(Type inputType, ParameterizedType type) {
-    if (type.isWildcard()) {
-      return true;
-    }
-    return inputType.accept(new IgnoreNullableAndParameters(type));
-  }
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: vbarua
body: REVIEW NOTE: this code is redundant with `isMatch(ParameterizedType actualType, ParameterizedType targetType) ` below
created_at: 2025-04-16T23:05:35Z
author_association: MEMBER
diff_hunk: @@ -411,53 +404,35 @@ private Optional<T> matchByLeastRestrictive(
         return Optional.empty();
       }
       Type type = typeConverter.toSubstrait(leastRestrictive);
-      var out = singularInputType.get().tryMatch(type, outputType);
-
-      if (out.isPresent()) {
-        var declaration = out.get();
-        var coercedArgs = coerceArguments(operands, type);
-        declaration.validateOutputType(coercedArgs, outputType);
-        return Optional.of(
-            generateBinding(
-                call,
-                out.get(),
-                coercedArgs.stream().map(FunctionArg.class::cast).collect(Collectors.toList()),
-                outputType));
-      }
-      return Optional.empty();
+      var out = singularInputType.orElseThrow().tryMatch(type, outputType);
path: isthmus/src/main/java/io/substrait/isthmus/expression/FunctionConverter.java
user: login: bestbeforetoday
body: Correct. No functional change by using `orElseThrow()`. The Java API documentation for `get()` says that `orElseThrow()` is preferred. I guess just for consistency with the other `orElse...` methods.
created_at: 2025-04-22T12:08:36Z
author_association: MEMBER
diff_hunk: @@ -65,6 +65,21 @@ public void inPredicate() throws IOException, SqlParseException {
             + "(SELECT L_SUPPKEY from LINEITEM where L_SUPPKEY < L_ORDERKEY)");
   }
 
+  @Test
+  public void concatStringLiteralAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select 'part_'||P_NAME from PART");
+  }
+
+  @Test
+  public void concatCharAndVarchar() throws Exception {
+    assertProtoPlanRoundrip("select P_BRAND||P_NAME from PART");
+  }
+
+  @Test
+  public void concatStringLiteralAndChar() throws Exception {
+    assertProtoPlanRoundrip("select 'brand_'||P_BRAND from PART");
+  }
path: isthmus/src/test/java/io/substrait/isthmus/ExpressionConvertabilityTest.java
user: login: bestbeforetoday
body: Done.
created_at: 2025-04-22T13:01:43Z
author_association: MEMBER

PR comments:
user: login: bestbeforetoday
created_at: 2025-03-06T10:33:20Z
body: Marking as draft until I can produce a better solution to the problems highlighted. I will mark for review again when ready.
author_association: MEMBER
user: login: bestbeforetoday
created_at: 2025-03-18T11:09:21Z
body: @vbarua Do you have any guidance on the best testing approach? The existing and added tests did not catch the not-quite-correct change I made previously, and it would be good for those tests to give better confidence that the conversion is working correctly.
author_association: MEMBER
user: login: bestbeforetoday
created_at: 2025-03-19T11:48:50Z
body: @vbarua I need some help untangling casting of arguments to the types required by a Substrait function. Given there is no exact match for the function signature, I can find a function to which I think I can coerce the supplied call. To do the coercion for operands that don't match the argument type required by the function, I need to create a cast. This requires the target `Type`. The `Function` can only give me the `ParameterizedType` of the arguments. Confusingly, `ParameterizedType` is a _superclass_ of `Type`, and the argument "types" I get from the `Function` are not instances of `Type`.

How do I get the `Type` values for a given `Function` arguments?
author_association: MEMBER
user: login: vbarua
created_at: 2025-03-20T05:51:55Z
body: > Do you have any guidance on the best testing approach? The existing and added tests did not catch the not-quite-correct change I made previously, and it would be good for those tests to give better confidence that the conversion is working correctly.

I don't have a specific approach in mind. It will likely requiere a combination of things. One thing that would be good to investigate would be adding further validation and/or validators for plans generally to catch issues like that.

> How do I get the Type values for a given Function arguments?

Unfortunately I'm not too familiar with that machinary. Someone in the Slack might know more.

Additionally, I wanted to add that adding cast is one way to solve this problem. There might be others, but it's what came to my mind first at least.
author_association: MEMBER
user: login: bestbeforetoday
created_at: 2025-03-20T16:58:30Z
body: @vbarua In the current (draft) iteration of the code I did some horrible, horrible ParameterizedType to Type mapping using visitors. This allows casts to be constructed for different string types, which at first glance look plausible. For example, this SQL:

```sql
select 'store' || s_store_name from store
```

generates Substrait including this (redacted) mapping:

```json
{
  "extensionUris": [{
    "extensionUriAnchor": 1,
    "uri": "/functions_string.yaml"
  }],
  "extensions": [{
    "extensionFunction": {
      "extensionUriReference": 1,
      "name": "concat:vchar"
    }
  }],
  "relations": [{
    "root": {
      "input": {
        "project": {
          [...]
          "expressions": [{
            "scalarFunction": {
              "outputType": {
                "varchar": {
                  "length": 55,
                  "nullability": "NULLABILITY_REQUIRED"
                }
              },
              "arguments": [{
                "value": {
                  "cast": {
                    "type": {
                      "varchar": {
                        "length": 5,
                        "nullability": "NULLABILITY_REQUIRED"
                      }
                    },
                    "input": {
                      "literal": {
                        "fixedChar": "store"
                      }
                    },
                    "failureBehavior": "FAILURE_BEHAVIOR_THROW_EXCEPTION"
                  }
                }
              }, {
                "value": {
                  "selection": {
                    "directReference": {
                      "structField": {
                        "field": 5
                      }
                    },
                    "rootReference": {
                    }
                  }
                }
              }]
            }
          }]
        }
      },
      "names": ["EXPR$0"]
    }
  }]
}
```

Unfortunately not all cases round-trip, and it breaks a few other test cases. If this looks to you like it is heading the right direction, I can try to get the failing test cases working again. What do you think?
author_association: MEMBER
user: login: vbarua
created_at: 2025-03-20T18:23:39Z
body: That does seem reasonable to me. One other thing to note is that `concat:vchar` is defined with a length param `L1`:
```
    impls:
      - args:
          - value: "varchar<L1>"
            name: "input"
        variadic:
          min: 1
        options:
          null_handling:
            values: [ IGNORE_NULLS, ACCEPT_NULLS ]
        return: "varchar<L1>"
```

As I understand the spec currently, if the output type of your `concat:vchar` declaration is `varchar<55>`, then all inputs must also be `varchar<55>`.

> I did some horrible, horrible ParameterizedType to Type mapping using visitors.

Another way to approach this might be to upcast the arguments in _Calcite_ before mapping into Substrait, though I don't actually know if there is anything for this within Calcite.

author_association: MEMBER
user: login: bestbeforetoday
created_at: 2025-04-22T13:17:13Z
body: @vbarua There are no longer any functionality changes to `FunctionConverter.java`. In previous iterations of this change there were but the type system change you suggested meant I could remove those changes. What is left is code refactor / clean-up that helped me to figure out what was actually happening, and to facilitate the (now removed) functional changes.

I have rebased on the latest main branch state and moved unit tests to the new `FunctionConversionTest`. I think all the review comments are addressed and this is ready for re-review. Please let me know if I missed anything.
author_association: MEMBER
user: login: vbarua
created_at: 2025-04-22T16:57:49Z
body: @bestbeforetoday thanks for working on this.

In the future, it would be helpful to call out that changes are mostly refactors and/or split them into their own PRs. The FunctionConverter code is pretty hairy, even for me, so I kept trying to find a time when I was mentally fresh to review them, but it turned out they were easy to reviews and I could have done it anytime really.
author_association: MEMBER
user: login: bestbeforetoday
created_at: 2025-04-23T13:41:59Z
body: @vbarua Are you able to merge this change so I can safely start additional changes to improve TPC-DS compatibility on top of it?
author_association: MEMBER
user: login: vbarua
created_at: 2025-04-23T14:56:14Z
body: @bestbeforetoday thanks for the reminder. Merged.
author_association: MEMBER