Repository: trinodb/trino
Pull request #24055: Fix case-sensitivity issue with views and column masks
Author: login: ksobolew
Labels: name: cla-signed
Created: 2024-11-07T08:42:31Z | Merged: 2025-03-06T13:06:46Z

<!-- Thank you for submitting a pull request! Find more information 
at https://trino.io/development/process.html, 
at https://github.com/trinodb/trino/blob/master/.github/DEVELOPMENT.md 
and contact us on #core-dev in Slack. -->
<!-- Provide an overview for maintainers and reviewers. -->
## Description

The table column reference was registered incorrectly with the original case taken from the view definition. It then failed to match the column schema returned from `SystemAccessControl` and the mask was not applied.

Instead of sprinkling `toLowerCase()` here and there, we will associate the original `Field` with the column mask and use `Field#canResove` to do the matching. The problem with this is that there's no way to do efficient lookups by name in a case-insensitive way, so we have to iterate the list of `Field`-`Expression` pairs to find a match.

Fixes #24054.

<!-- Provide details that help an engineer who is unfamiliar with this part of the code. -->
## Additional context and related issues

I wonder how many more issues like that will we find...

<!-- Mark the appropriate option with an (x). Propose a release note if you can.
More info at https://trino.io/development/process#release-note -->
## Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

```markdown
## Section
* Fixed column masks not applying on views declaring non-lowercase column names. ({issue}`24054`)
```


Linked issue #24054: Column masks don't work when view declares columns with uppercase names
The view can be as simple as something like this:

```sql
CREATE VIEW test_view AS SELECT 123 AS "COLUMN";
```

When there is a column mask on "column", it should work, because in Trino column names are case-insensitive, but it does not.

(It's difficult to have a full reproducer in a ticket description, because there's no syntax to create column masks in SQL.)

Diff:
diff --git a/core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java b/core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
index f2dd8027a4cc..47a117bda563 100644
--- a/core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
+++ b/core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
@@ -110,6 +110,7 @@
 import java.util.Optional;
 import java.util.OptionalLong;
 import java.util.Set;
+import java.util.stream.Stream;
 
 import static com.google.common.base.MoreObjects.toStringHelper;
 import static com.google.common.base.Preconditions.checkArgument;
@@ -118,6 +119,7 @@
 import static com.google.common.collect.ImmutableList.toImmutableList;
 import static com.google.common.collect.ImmutableMap.toImmutableMap;
 import static com.google.common.collect.ImmutableSet.toImmutableSet;
+import static com.google.common.collect.MoreCollectors.toOptional;
 import static io.trino.sql.analyzer.QueryType.DESCRIBE;
 import static io.trino.sql.analyzer.QueryType.EXPLAIN;
 import static java.lang.Boolean.FALSE;
@@ -232,7 +234,7 @@ public class Analysis
     private final Map<NodeRef<Table>, List<Expression>> checkConstraints = new LinkedHashMap<>();
 
     private final Multiset<ColumnMaskScopeEntry> columnMaskScopes = HashMultiset.create();
-    private final Map<NodeRef<Table>, Map<String, Expression>> columnMasks = new LinkedHashMap<>();
+    private final Map<NodeRef<Table>, Map<Field, Expression>> columnMasks = new LinkedHashMap<>();
 
     private final Map<NodeRef<Unnest>, UnnestAnalysis> unnestAnalysis = new LinkedHashMap<>();
     private Optional<Create> create = Optional.empty();
@@ -1161,15 +1163,15 @@ public void unregisterTableForColumnMasking(QualifiedObjectName table, String co
         referenceChain.pop();
     }
 
-    public void addColumnMask(Table table, String column, Expression mask)
+    public void addColumnMask(Table table, Field column, Expression mask)
     {
-        Map<String, Expression> masks = columnMasks.computeIfAbsent(NodeRef.of(table), node -> new LinkedHashMap<>());
-        checkArgument(!masks.containsKey(column), "Mask already exists for column %s", column);
+        Map<Field, Expression> masks = columnMasks.computeIfAbsent(NodeRef.of(table), _ -> new LinkedHashMap<>());
+        checkArgument(!masks.containsKey(column), "Mask already exists for column %s", column.getName().orElse("(unknown)"));
 
         masks.put(column, mask);
     }
 
-    public Map<String, Expression> getColumnMasks(Table table)
+    public Map<Field, Expression> getColumnMasks(Table table)
     {
         return unmodifiableMap(columnMasks.getOrDefault(NodeRef.of(table), ImmutableMap.of()));
     }
@@ -1189,7 +1191,7 @@ public List<TableInfo> getReferencedTables()
                             .distinct()
                             .map(fieldName -> new ColumnInfo(
                                     fieldName,
-                                    Optional.ofNullable(columnMasks.getOrDefault(table, ImmutableMap.of()).get(fieldName))
+                                    resolveColumnMask(table.getNode().getName(), fieldName, columnMasks.getOrDefault(table, ImmutableMap.of()))
                                             .map(Expression::toString)))
                             .collect(toImmutableList());
 
@@ -1210,6 +1212,23 @@ public List<TableInfo> getReferencedTables()
                 .collect(toImmutableList());
     }
 
+    private static Optional<Expression> resolveColumnMask(QualifiedName tableName, String fieldName, Map<Field, Expression> expressions)
+    {
+        QualifiedName qualifiedFieldName = concatIdentifier(tableName, fieldName);
+        return expressions.entrySet().stream()
+                .filter(fieldExpression -> fieldExpression.getKey().canResolve(qualifiedFieldName))
+                .collect(toOptional())
+                .map(Map.Entry::getValue);
+    }
+
+    private static QualifiedName concatIdentifier(QualifiedName tableName, String fieldName)
+    {
+        return QualifiedName.of(Stream.concat(
+                        tableName.getOriginalParts().stream(),
+                        Stream.of(new Identifier(fieldName)))
+                .collect(toImmutableList()));
+    }
+
     public List<RoutineInfo> getRoutines()
     {
         return resolvedFunctions.values().stream()
diff --git a/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java b/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
index 7f119f553f5e..33de0fcc09c6 100644
--- a/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
+++ b/core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
@@ -2370,23 +2370,33 @@ private void checkStorageTableNotRedirected(QualifiedObjectName source)
 
         private void analyzeFiltersAndMasks(Table table, QualifiedObjectName name, RelationType relationType, Scope accessControlScope)
         {
+            ImmutableList.Builder<Field> fieldBuilder = ImmutableList.builder();
             ImmutableList.Builder<ColumnSchema> columnSchemaBuilder = ImmutableList.builder();
             for (int index = 0; index < relationType.getAllFieldCount(); index++) {
                 Field field = relationType.getFieldByIndex(index);
-                field.getName().ifPresent(fieldName -> columnSchemaBuilder.add(ColumnSchema.builder()
-                        .setName(fieldName)
-                        .setType(field.getType())
-                        .setHidden(field.isHidden())
-                        .build()));
+                field.getName().ifPresent(fieldName -> {
+                    fieldBuilder.add(field);
+                    columnSchemaBuilder.add(ColumnSchema.builder()
+                            .setName(fieldName)
+                            .setType(field.getType())
+                            .setHidden(field.isHidden())
+                            .build());
+                });
             }
+            List<Field> fields = fieldBuilder.build();
             List<ColumnSchema> columnSchemas = columnSchemaBuilder.build();
 
             Map<ColumnSchema, ViewExpression> masks = accessControl.getColumnMasks(session.toSecurityContext(), name, columnSchemas);
 
-            for (ColumnSchema columnSchema : columnSchemas) {
+            for (Field field : fields) {
+                ColumnSchema columnSchema = ColumnSchema.builder()
+                        .setName(field.getName().orElseThrow())
+                        .setType(field.getType())
+                        .setHidden(field.isHidden())
+                        .build();
                 Optional.ofNullable(masks.get(columnSchema)).ifPresent(mask -> {
                     if (checkCanSelectFromColumn(name, columnSchema.getName())) {
-                        analyzeColumnMask(session.getIdentity().getUser(), table, name, columnSchema, accessControlScope, mask);
+                        analyzeColumnMask(session.getIdentity().getUser(), table, name, field, accessControlScope, mask);
                     }
                 });
             }
@@ -5215,9 +5225,9 @@ private void analyzeCheckConstraint(Table table, QualifiedObjectName name, Scope
             analysis.addCheckConstraints(table, expression);
         }
 
-        private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObjectName tableName, ColumnSchema columnSchema, Scope scope, ViewExpression mask)
+        private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObjectName tableName, Field field, Scope scope, ViewExpression mask)
         {
-            String column = columnSchema.getName();
+            String column = field.getName().orElseThrow();
             if (analysis.hasColumnMask(tableName, column, currentIdentity)) {
                 throw new TrinoException(INVALID_COLUMN_MASK, extractLocation(table), format("Column mask for '%s.%s' is recursive", tableName, column), null);
             }
@@ -5261,13 +5271,13 @@ private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObj
 
             analysis.recordSubqueries(expression, expressionAnalysis);
 
-            Type expectedType = columnSchema.getType();
+            Type expectedType = field.getType();
             Type actualType = expressionAnalysis.getType(expression);
             if (!actualType.equals(expectedType)) {
                 TypeCoercion coercion = new TypeCoercion(plannerContext.getTypeManager()::getType);
 
-                if (!coercion.canCoerce(actualType, columnSchema.getType())) {
-                    throw new TrinoException(TYPE_MISMATCH, extractLocation(table), format("Expected column mask for '%s.%s' to be of type %s, but was %s", tableName, column, columnSchema.getType(), actualType), null);
+                if (!coercion.canCoerce(actualType, field.getType())) {
+                    throw new TrinoException(TYPE_MISMATCH, extractLocation(table), format("Expected column mask for '%s.%s' to be of type %s, but was %s", tableName, column, field.getType(), actualType), null);
                 }
 
                 // TODO: this should be "coercion.isTypeOnlyCoercion(actualType, expectedType)", but type-only coercions are broken
@@ -5277,7 +5287,7 @@ private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObj
                 analysis.addCoercion(expression, expectedType);
             }
 
-            analysis.addColumnMask(table, column, expression);
+            analysis.addColumnMask(table, field, expression);
         }
 
         private List<Expression> descriptorToFields(Scope scope)
diff --git a/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java b/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
index 1db38dfa13b4..2a6610c800d3 100644
--- a/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
+++ b/core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
@@ -422,7 +422,7 @@ public RelationPlan addCheckConstraints(List<io.trino.sql.tree.Expression> const
 
     private RelationPlan addColumnMasks(Table table, RelationPlan plan)
     {
-        Map<String, io.trino.sql.tree.Expression> columnMasks = analysis.getColumnMasks(table);
+        Map<Field, io.trino.sql.tree.Expression> columnMasks = analysis.getColumnMasks(table);
 
         // A Table can represent a WITH query, which can have anonymous fields. On the other hand,
         // it can't have masks. The loop below expects fields to have proper names, so bail out
@@ -441,7 +441,7 @@ private RelationPlan addColumnMasks(Table table, RelationPlan plan)
         for (int i = 0; i < plan.getDescriptor().getAllFieldCount(); i++) {
             Field field = plan.getDescriptor().getFieldByIndex(i);
 
-            io.trino.sql.tree.Expression mask = columnMasks.get(field.getName().orElseThrow());
+            io.trino.sql.tree.Expression mask = columnMasks.get(field);
             Symbol symbol = plan.getFieldMappings().get(i);
             Expression projection = symbol.toSymbolReference();
             if (mask != null) {
diff --git a/core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java b/core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
index e34e11f9d23b..560faad79a9f 100644
--- a/core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
+++ b/core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
@@ -88,6 +88,17 @@ public TestColumnMask()
                 Optional.of(VIEW_OWNER),
                 false,
                 ImmutableList.of());
+        ConnectorViewDefinition viewUpperCase = new ConnectorViewDefinition(
+                "SELECT NATIONKEY, NAME FROM local.tiny.nation",
+                Optional.empty(),
+                Optional.empty(),
+                ImmutableList.of(
+                        new ConnectorViewDefinition.ViewColumn("NATIONKEY", BigintType.BIGINT.getTypeId(), Optional.empty()),
+                        new ConnectorViewDefinition.ViewColumn("NAME", VarcharType.createVarcharType(25).getTypeId(), Optional.empty())),
+                Optional.empty(),
+                Optional.of(VIEW_OWNER),
+                false,
+                ImmutableList.of());
 
         ConnectorViewDefinition viewWithNested = new ConnectorViewDefinition(
                 """
@@ -170,6 +181,7 @@ public TestColumnMask()
                 })
                 .withGetViews((s, prefix) -> ImmutableMap.of(
                         new SchemaTableName("default", "nation_view"), view,
+                        new SchemaTableName("default", "nation_view_uppercase"), viewUpperCase,
                         new SchemaTableName("default", "view_with_nested"), viewWithNested))
                 .withGetMaterializedViews((s, prefix) -> ImmutableMap.of(
                         new SchemaTableName("default", "nation_materialized_view"), materializedView,
@@ -455,6 +467,23 @@ public void testView()
         assertThat(assertions.query("SELECT name FROM mock.default.nation_view WHERE nationkey = 1")).matches("VALUES CAST('ANITNEGRA' AS VARCHAR(25))");
     }
 
+    @Test
+    public void testViewWithUppercaseColumnName()
+    {
+        accessControl.reset();
+        accessControl.columnMask(
+                new QualifiedObjectName(MOCK_CATALOG, "default", "nation_view_uppercase"),
+                "name",
+                USER,
+                ViewExpression.builder()
+                        .identity(USER)
+                        .catalog(LOCAL_CATALOG)
+                        .schema("tiny")
+                        .expression("reverse(name)")
+                        .build());
+        assertThat(assertions.query("SELECT name FROM mock.default.nation_view_uppercase WHERE nationkey = 1")).matches("VALUES CAST('ANITNEGRA' AS VARCHAR(25))");
+    }
+
     @Test
     public void testTableReferenceInWithClause()
     {


Reviews:
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-11-07T08:44:51Z
user: login: martint
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-11-08T11:16:27Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-11-14T13:38:42Z
user: login: homar
state: APPROVED
author_association: MEMBER
submitted_at: 2024-11-26T10:18:46Z
user: login: martint
state: COMMENTED
author_association: MEMBER
submitted_at: 2024-12-15T19:08:24Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-12-18T10:21:57Z
user: login: kokosing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-07T11:20:07Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-07T11:55:32Z
user: login: kokosing
body: % comments
state: APPROVED
author_association: MEMBER
submitted_at: 2025-02-10T13:44:38Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-11T10:21:03Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-11T10:22:41Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-11T10:25:09Z
user: login: martint
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-11T13:51:49Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-12T11:41:28Z
user: login: martint
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-12T14:15:08Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-14T11:24:52Z
user: login: kokosing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T12:50:28Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T13:18:50Z
user: login: kokosing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:53:29Z
user: login: ksobolew
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T14:45:33Z
user: login: kokosing
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-04T10:36:57Z

Review comments:
diff_hunk: @@ -766,7 +766,7 @@ private Type handleResolvedField(Expression node, ResolvedField resolvedField, C
             }
 
             if (field.getOriginTable().isPresent() && field.getOriginColumnName().isPresent()) {
-                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get());
+                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get().toLowerCase(ENGLISH));
path: core/trino-main/src/main/java/io/trino/sql/analyzer/ExpressionAnalyzer.java
user: login: ksobolew
body: @martint I sprinkled those `toLowerCase` in enough places to fix this particular issue, but I'm not entirely sure if these are the right places. I think the convention is to keep the original case in the AST and only convert it as needed in the analyzer, even though it would probably be easier to fix all issues like this by storing converted names in AST. WDYT?
created_at: 2024-11-07T08:44:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -766,7 +766,7 @@ private Type handleResolvedField(Expression node, ResolvedField resolvedField, C
             }
 
             if (field.getOriginTable().isPresent() && field.getOriginColumnName().isPresent()) {
-                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get());
+                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get().toLowerCase(ENGLISH));
path: core/trino-main/src/main/java/io/trino/sql/analyzer/ExpressionAnalyzer.java
user: login: martint
body: Identifiers need to be resolved according to standard sql rules. It differs for quoted vs non-quoted identifiers.

Currently, there are some inconsistencies  in how they are being handled in various places, but we have a long standing issue to fix them. See https://github.com/trinodb/trino/issues/17
created_at: 2024-11-08T11:16:27Z
author_association: MEMBER
diff_hunk: @@ -766,7 +766,7 @@ private Type handleResolvedField(Expression node, ResolvedField resolvedField, C
             }
 
             if (field.getOriginTable().isPresent() && field.getOriginColumnName().isPresent()) {
-                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get());
+                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get().toLowerCase(ENGLISH));
path: core/trino-main/src/main/java/io/trino/sql/analyzer/ExpressionAnalyzer.java
user: login: ksobolew
body: Understood @martint. The question is, though, how far this should extend to interfaces like access control. This is especially relevant to what I call "negative grants" - column masks, row filters and DENY policies. If an access control declares a column mask on column `foo`, it should apply to `foo` and `Foo` but not to `"Foo"`, if I understand correctly. One could argue that this is unexpected and may cause information leaks. To counter that someone could say that if the user uses case-sensitive names then they know what they are doing. But to counter *that*, some access controls coerce all names to lower case, assuming that they will be matched case-insensitively. So I am not sure what to do next.

I believe this needs to be fixed somehow right now, because at this moment this enables accidental information leaks, like described above, but with regular identifiers.
created_at: 2024-11-14T13:38:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -766,7 +766,7 @@ private Type handleResolvedField(Expression node, ResolvedField resolvedField, C
             }
 
             if (field.getOriginTable().isPresent() && field.getOriginColumnName().isPresent()) {
-                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get());
+                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get().toLowerCase(ENGLISH));
path: core/trino-main/src/main/java/io/trino/sql/analyzer/ExpressionAnalyzer.java
user: login: martint
body: I think we can fix this by associating masks to the corresponding Field in StatementAnalyzer and Analysis, instead of doing it by name. Then, in RelationPlanner, the lookup can be done directly by Field instead of relying on the name having the right case.
created_at: 2024-12-15T19:08:24Z
author_association: MEMBER
diff_hunk: @@ -766,7 +766,7 @@ private Type handleResolvedField(Expression node, ResolvedField resolvedField, C
             }
 
             if (field.getOriginTable().isPresent() && field.getOriginColumnName().isPresent()) {
-                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get());
+                tableColumnReferences.put(field.getOriginTable().get(), field.getOriginColumnName().get().toLowerCase(ENGLISH));
path: core/trino-main/src/main/java/io/trino/sql/analyzer/ExpressionAnalyzer.java
user: login: ksobolew
body: @martint I pushed a prototype of a solution which (I think) implements your suggestion. There's no way to do efficient lookups of `Field`s, though, so in some places we need to do a linear scan of the list of masks in order to find a match.
created_at: 2024-12-18T10:21:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -171,6 +182,7 @@ public TestColumnMask()
                 })
                 .withGetViews((s, prefix) -> ImmutableMap.of(
                         new SchemaTableName("default", "nation_view"), view,
+                        new SchemaTableName("default", "nation_view_uppercase"), viewWithDifferentCase,
path: core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
user: login: kokosing
body: uppercase and differentcase? I guess you want to pick one.
created_at: 2025-02-07T11:15:32Z
author_association: MEMBER
diff_hunk: @@ -89,6 +89,17 @@ public TestColumnMask()
                 Optional.of(VIEW_OWNER),
                 false,
                 ImmutableList.of());
+        ConnectorViewDefinition viewWithDifferentCase = new ConnectorViewDefinition(
path: core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
user: login: kokosing
body: viewWithUpperCasedColumnNames
created_at: 2025-02-07T11:16:09Z
author_association: MEMBER
diff_hunk: @@ -441,10 +441,11 @@ private RelationPlan addColumnMasks(Table table, RelationPlan plan)
         for (int i = 0; i < plan.getDescriptor().getAllFieldCount(); i++) {
             Field field = plan.getDescriptor().getFieldByIndex(i);
 
-            io.trino.sql.tree.Expression mask = columnMasks.get(field.getName().orElseThrow());
+            Optional<io.trino.sql.tree.Expression> columnMask = Analysis.resolveColumnMask(table.getName(), field.getName().orElseThrow(), columnMasks);
path: core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
user: login: kokosing
body: I don't understand this. You pass multiple columns (field expressions) and you get a single mask. I think you should pass only a single column here.
created_at: 2025-02-07T11:20:00Z
author_association: MEMBER
diff_hunk: @@ -441,10 +441,11 @@ private RelationPlan addColumnMasks(Table table, RelationPlan plan)
         for (int i = 0; i < plan.getDescriptor().getAllFieldCount(); i++) {
             Field field = plan.getDescriptor().getFieldByIndex(i);
 
-            io.trino.sql.tree.Expression mask = columnMasks.get(field.getName().orElseThrow());
+            Optional<io.trino.sql.tree.Expression> columnMask = Analysis.resolveColumnMask(table.getName(), field.getName().orElseThrow(), columnMasks);
path: core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
user: login: ksobolew
body: The point here is that we have to pick one matching mask from all masks that we got for the table. We used to do it via a `Map` lookup by column name, but with delimited identifiers we cannot do a simple thing like that.
created_at: 2025-02-07T11:55:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2377,23 +2377,33 @@ private void checkStorageTableNotRedirected(QualifiedObjectName source)
 
         private void analyzeFiltersAndMasks(Table table, QualifiedObjectName name, RelationType relationType, Scope accessControlScope)
         {
+            ImmutableList.Builder<Field> fieldBuilder = ImmutableList.builder();
             ImmutableList.Builder<ColumnSchema> columnSchemaBuilder = ImmutableList.builder();
             for (int index = 0; index < relationType.getAllFieldCount(); index++) {
                 Field field = relationType.getFieldByIndex(index);
-                field.getName().ifPresent(fieldName -> columnSchemaBuilder.add(ColumnSchema.builder()
-                        .setName(fieldName)
-                        .setType(field.getType())
-                        .setHidden(field.isHidden())
-                        .build()));
+                field.getName().ifPresent(fieldName -> {
+                    fieldBuilder.add(field);
+                    columnSchemaBuilder.add(ColumnSchema.builder()
+                            .setName(fieldName)
+                            .setType(field.getType())
+                            .setHidden(field.isHidden())
+                            .build());
+                });
             }
+            List<Field> fields = fieldBuilder.build();
             List<ColumnSchema> columnSchemas = columnSchemaBuilder.build();
path: core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
user: login: kokosing
body: is this still used?
created_at: 2025-02-10T13:41:56Z
author_association: MEMBER
diff_hunk: @@ -5222,9 +5232,9 @@ private void analyzeCheckConstraint(Table table, QualifiedObjectName name, Scope
             analysis.addCheckConstraints(table, expression);
         }
 
-        private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObjectName tableName, ColumnSchema columnSchema, Scope scope, ViewExpression mask)
+        private void analyzeColumnMask(String currentIdentity, Table table, QualifiedObjectName tableName, Field columnSchema, Scope scope, ViewExpression mask)
path: core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
user: login: kokosing
body: rename `columnSchema` to field, or restore previous type..
created_at: 2025-02-10T13:42:35Z
author_association: MEMBER
diff_hunk: @@ -89,6 +89,17 @@ public TestColumnMask()
                 Optional.of(VIEW_OWNER),
                 false,
                 ImmutableList.of());
+        ConnectorViewDefinition viewWithDifferentCase = new ConnectorViewDefinition(
path: core/trino-main/src/test/java/io/trino/sql/query/TestColumnMask.java
user: login: ksobolew
body: I'll go with just `viewUpperCase`. The database objects are `nation_view` and `nation_view_uppercase` so the Java variables can be `view` and `viewUpperCase`.
created_at: 2025-02-11T10:21:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2377,23 +2377,33 @@ private void checkStorageTableNotRedirected(QualifiedObjectName source)
 
         private void analyzeFiltersAndMasks(Table table, QualifiedObjectName name, RelationType relationType, Scope accessControlScope)
         {
+            ImmutableList.Builder<Field> fieldBuilder = ImmutableList.builder();
             ImmutableList.Builder<ColumnSchema> columnSchemaBuilder = ImmutableList.builder();
             for (int index = 0; index < relationType.getAllFieldCount(); index++) {
                 Field field = relationType.getFieldByIndex(index);
-                field.getName().ifPresent(fieldName -> columnSchemaBuilder.add(ColumnSchema.builder()
-                        .setName(fieldName)
-                        .setType(field.getType())
-                        .setHidden(field.isHidden())
-                        .build()));
+                field.getName().ifPresent(fieldName -> {
+                    fieldBuilder.add(field);
+                    columnSchemaBuilder.add(ColumnSchema.builder()
+                            .setName(fieldName)
+                            .setType(field.getType())
+                            .setHidden(field.isHidden())
+                            .build());
+                });
             }
+            List<Field> fields = fieldBuilder.build();
             List<ColumnSchema> columnSchemas = columnSchemaBuilder.build();
path: core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
user: login: ksobolew
body: Yes, `columnSchemas` is what we pass to access control.
created_at: 2025-02-11T10:22:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2377,23 +2377,33 @@ private void checkStorageTableNotRedirected(QualifiedObjectName source)
 
         private void analyzeFiltersAndMasks(Table table, QualifiedObjectName name, RelationType relationType, Scope accessControlScope)
         {
+            ImmutableList.Builder<Field> fieldBuilder = ImmutableList.builder();
             ImmutableList.Builder<ColumnSchema> columnSchemaBuilder = ImmutableList.builder();
             for (int index = 0; index < relationType.getAllFieldCount(); index++) {
                 Field field = relationType.getFieldByIndex(index);
-                field.getName().ifPresent(fieldName -> columnSchemaBuilder.add(ColumnSchema.builder()
-                        .setName(fieldName)
-                        .setType(field.getType())
-                        .setHidden(field.isHidden())
-                        .build()));
+                field.getName().ifPresent(fieldName -> {
+                    fieldBuilder.add(field);
+                    columnSchemaBuilder.add(ColumnSchema.builder()
+                            .setName(fieldName)
+                            .setType(field.getType())
+                            .setHidden(field.isHidden())
+                            .build());
+                });
             }
+            List<Field> fields = fieldBuilder.build();
             List<ColumnSchema> columnSchemas = columnSchemaBuilder.build();
path: core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java
user: login: ksobolew
body: But note that the SPI effectively forces everything to be lower-case for access control.
created_at: 2025-02-11T10:25:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -441,10 +441,11 @@ private RelationPlan addColumnMasks(Table table, RelationPlan plan)
         for (int i = 0; i < plan.getDescriptor().getAllFieldCount(); i++) {
             Field field = plan.getDescriptor().getFieldByIndex(i);
 
-            io.trino.sql.tree.Expression mask = columnMasks.get(field.getName().orElseThrow());
+            Optional<io.trino.sql.tree.Expression> columnMask = Analysis.resolveColumnMask(table.getName(), field.getName().orElseThrow(), columnMasks);
path: core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
user: login: martint
body: This shouldn't be doing any resolution. It has the `Field`, so all it needs to do is look up the corresponding expression for that field, which should be stored in the analysis object.

So, in a nutshell:
* The analyzer resolves the schema associated with the underlying table (modeled by RelationType)
* The analyzer resolves the masks by column name and analyzes them (type checks, etc). It associates them with the corresponding Field from RelationType.
* The planner injects the necessary projections for each Field by looking them up in the data structure produces by the analyzer (Analysis)
created_at: 2025-02-11T13:51:49Z
author_association: MEMBER
diff_hunk: @@ -441,10 +441,11 @@ private RelationPlan addColumnMasks(Table table, RelationPlan plan)
         for (int i = 0; i < plan.getDescriptor().getAllFieldCount(); i++) {
             Field field = plan.getDescriptor().getFieldByIndex(i);
 
-            io.trino.sql.tree.Expression mask = columnMasks.get(field.getName().orElseThrow());
+            Optional<io.trino.sql.tree.Expression> columnMask = Analysis.resolveColumnMask(table.getName(), field.getName().orElseThrow(), columnMasks);
path: core/trino-main/src/main/java/io/trino/sql/planner/RelationPlanner.java
user: login: ksobolew
body: Thanks! I can see there are already `Map`s where `Field` is the key - I wasn't sure I can use it this way, but now the `columnMasks` collection simply changes from `Map<String, Expression>` to `Map<Field, Expression>` and it simplifies things a bit.
created_at: 2025-02-12T11:41:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1189,7 +1190,7 @@ public List<TableInfo> getReferencedTables()
                             .distinct()
                             .map(fieldName -> new ColumnInfo(
                                     fieldName,
-                                    Optional.ofNullable(columnMasks.getOrDefault(table, ImmutableMap.of()).get(fieldName))
+                                    resolveColumnMask(table.getNode().getName(), fieldName, columnMasks.getOrDefault(table, ImmutableMap.of()))
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: martint
body: This should be just a lookup in a map. The `Analysis` object should not be doing any resolution -- that should've already been done during the analysis process. You'll need to adjust the value of `columnMasks` to be a `Map<Field, Expression>`.
created_at: 2025-02-12T14:15:08Z
author_association: MEMBER
diff_hunk: @@ -1189,7 +1190,7 @@ public List<TableInfo> getReferencedTables()
                             .distinct()
                             .map(fieldName -> new ColumnInfo(
                                     fieldName,
-                                    Optional.ofNullable(columnMasks.getOrDefault(table, ImmutableMap.of()).get(fieldName))
+                                    resolveColumnMask(table.getNode().getName(), fieldName, columnMasks.getOrDefault(table, ImmutableMap.of()))
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: ksobolew
body: `columnMasks` already is adjusted. What I would have to adjust is `Analysis#tableColumnReferences`, which currently stores column names as `String` (and uses this to deduplicate references, I think?).
created_at: 2025-02-14T11:24:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1210,6 +1211,22 @@ public List<TableInfo> getReferencedTables()
                 .collect(toImmutableList());
     }
 
+    private static Optional<Expression> resolveColumnMask(QualifiedName tableName, String fieldName, Map<Field, Expression> expressions)
+    {
+        return expressions.entrySet().stream()
+                .filter(fieldExpression -> fieldExpression.getKey().canResolve(concatIdentifier(tableName, fieldName)))
+                .findFirst()
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: kokosing
body: Is it ok to have multiple matches here? Shouldn't we expect `Optional` semantic here?
created_at: 2025-03-03T12:49:49Z
author_association: MEMBER
diff_hunk: @@ -1210,6 +1211,22 @@ public List<TableInfo> getReferencedTables()
                 .collect(toImmutableList());
     }
 
+    private static Optional<Expression> resolveColumnMask(QualifiedName tableName, String fieldName, Map<Field, Expression> expressions)
+    {
+        return expressions.entrySet().stream()
+                .filter(fieldExpression -> fieldExpression.getKey().canResolve(concatIdentifier(tableName, fieldName)))
+                .findFirst()
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: ksobolew
body: This is a good question. I think it's impossible to have multiple matches here. Like @martint says, ideally this would be a `Map` lookup, this should be a 1:0..1 relationship here.
created_at: 2025-03-03T13:18:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1210,6 +1211,22 @@ public List<TableInfo> getReferencedTables()
                 .collect(toImmutableList());
     }
 
+    private static Optional<Expression> resolveColumnMask(QualifiedName tableName, String fieldName, Map<Field, Expression> expressions)
+    {
+        return expressions.entrySet().stream()
+                .filter(fieldExpression -> fieldExpression.getKey().canResolve(concatIdentifier(tableName, fieldName)))
+                .findFirst()
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: kokosing
body: So let's use `.collect(toOptional())`
created_at: 2025-03-03T13:53:29Z
author_association: MEMBER
diff_hunk: @@ -1210,6 +1211,22 @@ public List<TableInfo> getReferencedTables()
                 .collect(toImmutableList());
     }
 
+    private static Optional<Expression> resolveColumnMask(QualifiedName tableName, String fieldName, Map<Field, Expression> expressions)
+    {
+        return expressions.entrySet().stream()
+                .filter(fieldExpression -> fieldExpression.getKey().canResolve(concatIdentifier(tableName, fieldName)))
+                .findFirst()
path: core/trino-main/src/main/java/io/trino/sql/analyzer/Analysis.java
user: login: ksobolew
body: Done
created_at: 2025-03-03T14:45:33Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-01-08T17:03:11Z
body: This pull request has gone a while without any activity. Tagging for triage help: @mosabua
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-01-30T17:03:07Z
body: Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.
author_association: NONE
user: login: kokosing
created_at: 2025-02-11T09:30:24Z
body: @martint it looks like we are taking some tradeoff here. We are doing a linear check for column masks. I assume it is fine because I don't imagine having hundreds of column masks per table, and also it is a good step anyway as we here improve correctness. We can optimize it later if we notice that is a real problem. I don't want do prematurely optimize things.
author_association: MEMBER
user: login: kokosing
created_at: 2025-03-04T10:38:47Z
body: @martint it looks like all the comments are addressed. I would like to move forward with this as this is a fixing a bug, please let me know if you have any more comments, otherwise I would like to merge it.
author_association: MEMBER
user: login: kokosing
created_at: 2025-03-06T13:06:34Z
body: This PR is fixing the bug, so I think it is important to get this in. If there are any follow up comments about the execution flow or structures I think @ksobolew will be happy to address them in another PR. So any post merge code review is welcome.
author_association: MEMBER
user: login: ksobolew
created_at: 2025-03-06T13:16:16Z
body: Thank you @kokosing!
author_association: CONTRIBUTOR