Repository: elastic/elasticsearch
Pull request #127563: ESQL: Avoid unintended attribute removal
Author: login: kanoshiou
Labels: name: >bug
name: Team:Analytics
description: Meta label for analytical engine team (ESQL/Aggs/Geo)
name: external-contributor
description: Pull request authored by a developer outside the Elasticsearch team
name: auto-backport
description: Automatically create backport pull requests when merged
name: :Analytics/ES|QL
description: AKA ESQL
name: v8.19.0
name: v9.1.0
name: v9.0.2
name: v8.18.4
Created: 2025-04-30T12:15:11Z | Merged: 2025-05-21T07:19:10Z

Because we aim to minimize the number of attributes required from `field_caps` at pre-analysis time, a removal check helps eliminate fields defined by users (such as `eval`). However, commands like `lookup join` and `enrich` can introduce fields with the same names as user-defined fields. To accommodate this, we ensure that aliases are not removed from the attributes list when a subsequent `JOIN` occurs.


For the query:
```
from message_types 
| eval type = 1 
| lookup join message_types_lookup on message 
| drop  message 
| grok type "%{WORD:b}" 
| stats x = max(b) 
| keep x
```

The plan `grok type "%{WORD:b}` passes the removal check and initiates alias removal down the tree. However, the removal process should halt at `lookup join` due to the reason mentioned above. The issue arises because the original code allows progression to `eval type = 1`, which inadvertently removes the `type` field. As a result, the field `type` ends up being of type `integer` defined by `eval type = 1`, rather than preserving the `keyword` type field `type` obtained from `message_types_lookup`.


Closes #127468

Linked issue #127468: ES|QL: Grok only supports KEYWORD or TEXT values, found expression [type] type [INTEGER]
With CSV dataset


```
from multivalue_points,h*,messa* 
| eval  `card` = true, PbehoQUqKSF = "VLGjhcgNkQiEVyCLo", DsxMWtGL = true, qSxTIvUorMim = true, `location` = 8593178066470220111, type = -446161601, FSkGQkgmS = false 
| eval  PbehoQUqKSF = 753987034, HLNMQfQj = true, `within` = true, `id` = "JDKKkYwhhh", lk = null, aecuvjTkgZza = 510616700, aDAMpuVtNX = null, qCopgNZPt = "AjhJUtZefqKdJYH", BxHHlFoA = "isBrmhKLc"
| rename message as message 
| lookup join message_types_lookup on message 
| sort PbehoQUqKSF DESC, ip1 DESC NULLS LAST 
| limit 5845 
| drop `subset`, ip*, `card`, `within`, host.v*, description, `aecuvjTkgZza`, host.version, `ip0`, height_range, DsxMWtGL, host_group, `aDAMpuVtNX`, PbehoQUqKSF, `intersects`, `host.os`, aDAMpuVtNX, *ight_range, HLNMQfQj, `FSkGQkgmS`, BxHHlFoA, card 
| grok type "%{WORD:GknCxQFo}" 
| eval  `location` = null, ZjWUUvGusyyz = null, HeeKIpzgh = false, `id` = 4325287503714500302, host = false, `lk` = null, HvTQdOqFajpH = false, fKNlsYoT = true, `location` = -1158449473, `qCopgNZPt` = 1219986202615280617 
| drop HeeKIpzg*, `ZjWUUvGusyyz`, `message`, `type`, `lk` 
| grok GknCxQFo "%{WORD:location} %{WORD:HvTQdOqFajpH}" 
| drop HvTQdOqFajpH, `location`, centroid 
| mv_expand GknCxQFo 
| limit 410 
| limit 3815 
| rename `id` AS `GknCxQFo` 
| grok host.name "%{WORD:oGQQZHxQHj} %{WORD:qCopgNZPt} %{WORD:vHKOmmocPcTO}" 
| stats  BkQXJRMeAM = min(GknCxQFo) 
| keep `BkQXJRMeAM`
```


```
verification_exceptionreason
Found 1 problem |
line 1:787: Grok only supports KEYWORD or TEXT values, found expression [type] type [INTEGER]
```
Linked issue comments: user: login: elasticsearchmachine
created_at: 2025-04-28T14:40:07Z
body: Pinging @elastic/es-analytical-engine (Team:Analytics)
author_association: COLLABORATOR
user: login: kanoshiou
created_at: 2025-04-30T07:18:17Z
body: Simplified query here:
```
from message_types 
| eval type = 1 
| lookup join message_types_lookup on message 
| drop  message 
| grok type "%{WORD:b}" 
| stats x = max(b) 
| keep x
```
author_association: CONTRIBUTOR

Diff:
diff --git a/docs/changelog/127563.yaml b/docs/changelog/127563.yaml
new file mode 100644
index 0000000000000..1e88e9d487b25
--- /dev/null
+++ b/docs/changelog/127563.yaml
@@ -0,0 +1,6 @@
+pr: 127563
+summary: "ESQL: Avoid unintended attribute removal"
+area: ES|QL
+type: bug
+issues:
+  - 127468
diff --git a/x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java b/x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java
index 9e07e8c162432..8ae28477e03bb 100644
--- a/x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java
+++ b/x-pack/plugin/esql/qa/server/src/main/java/org/elasticsearch/xpack/esql/qa/rest/generative/GenerativeRestTest.java
@@ -51,7 +51,6 @@ public abstract class GenerativeRestTest extends ESRestTestCase {
         "Unknown column \\[<all-fields-projected>\\]", // https://github.com/elastic/elasticsearch/issues/121741,
         "Plan \\[ProjectExec\\[\\[<no-fields>.* optimized incorrectly due to missing references", // https://github.com/elastic/elasticsearch/issues/125866
         "optimized incorrectly due to missing references", // https://github.com/elastic/elasticsearch/issues/116781
-        "only supports KEYWORD or TEXT values", // https://github.com/elastic/elasticsearch/issues/127468
         "The incoming YAML document exceeds the limit:" // still to investigate, but it seems to be specific to the test framework
     );
 
diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/dissect.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/dissect.csv-spec
index 2b3b0bee93471..f8704a20af8c3 100644
--- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/dissect.csv-spec
+++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/dissect.csv-spec
@@ -331,3 +331,43 @@ ROW a="b c d x"| DISSECT a "%{b} %{} %{d} %{}";
 a:keyword  | b:keyword | d:keyword
 b c d x    | b         | d
 ;
+
+avoidAttributesRemoval
+// https://github.com/elastic/elasticsearch/issues/127468
+required_capability: keep_regex_extract_attributes
+required_capability: join_lookup_v12
+from message_types 
+| eval type = 1 
+| lookup join message_types_lookup on message 
+| drop message 
+| dissect type "%{b}" 
+| stats x = max(b) 
+| keep x
+;
+
+x:keyword
+Success
+;
+
+avoidAttributesRemoval2
+// https://github.com/elastic/elasticsearch/issues/127468
+required_capability: keep_regex_extract_attributes
+required_capability: join_lookup_v12
+FROM sample_data, employees
+| EVAL client_ip = client_ip::keyword
+| RENAME languages AS language_code
+| LOOKUP JOIN clientips_lookup ON client_ip
+| EVAL type = 1::keyword
+| EVAL type = 2
+| LOOKUP JOIN message_types_lookup ON message
+| LOOKUP JOIN languages_lookup ON language_code
+| DISSECT type "%{type_as_text}"
+| KEEP message
+| WHERE message IS NOT NULL
+| SORT message DESC
+| LIMIT 1
+;
+
+message:keyword
+Disconnected
+;
diff --git a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/grok.csv-spec b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/grok.csv-spec
index 6dc9148ffc0e8..7fd9ed142f470 100644
--- a/x-pack/plugin/esql/qa/testFixtures/src/main/resources/grok.csv-spec
+++ b/x-pack/plugin/esql/qa/testFixtures/src/main/resources/grok.csv-spec
@@ -297,3 +297,35 @@ row text = "123 abc", int = 5 | sort int asc | grok text "%{NUMBER:text:int} %{W
 text:integer | int:integer | description:keyword
 123          | 5           | abc
 ;
+
+avoidAttributesRemoval
+// https://github.com/elastic/elasticsearch/issues/127468
+required_capability: union_types
+required_capability: join_lookup_v12
+required_capability: keep_regex_extract_attributes
+from multivalue_points,h*,messa* 
+| eval  `card` = true, PbehoQUqKSF = "VLGjhcgNkQiEVyCLo", DsxMWtGL = true, qSxTIvUorMim = true, `location` = 8593178066470220111, type = -446161601, FSkGQkgmS = false 
+| eval  PbehoQUqKSF = 753987034, HLNMQfQj = true, `within` = true, `id` = "JDKKkYwhhh", lk = null, aecuvjTkgZza = 510616700, aDAMpuVtNX = null, qCopgNZPt = "AjhJUtZefqKdJYH", BxHHlFoA = "isBrmhKLc"
+| rename message as message 
+| lookup join message_types_lookup on message 
+| sort PbehoQUqKSF DESC, ip1 DESC NULLS LAST 
+| limit 5845 
+| drop `subset`, ip*, `card`, `within`, host.v*, description, `aecuvjTkgZza`, host.version, `ip0`, height_range, DsxMWtGL, host_group, `aDAMpuVtNX`, PbehoQUqKSF, `intersects`, `host.os`, aDAMpuVtNX, *ight_range, HLNMQfQj, `FSkGQkgmS`, BxHHlFoA, card 
+| grok type "%{WORD:GknCxQFo}" 
+| eval  `location` = null, ZjWUUvGusyyz = null, HeeKIpzgh = false, `id` = 4325287503714500302, host = false, `lk` = null, HvTQdOqFajpH = false, fKNlsYoT = true, `location` = -1158449473, `qCopgNZPt` = 1219986202615280617 
+| drop HeeKIpzg*, `ZjWUUvGusyyz`, `message`, `type`, `lk` 
+| grok GknCxQFo "%{WORD:location} %{WORD:HvTQdOqFajpH}" 
+| drop HvTQdOqFajpH, `location`, centroid 
+| mv_expand GknCxQFo 
+| limit 410 
+| limit 3815 
+| rename `id` AS `GknCxQFo` 
+| grok host.name "%{WORD:oGQQZHxQHj} %{WORD:qCopgNZPt} %{WORD:vHKOmmocPcTO}" 
+| stats  BkQXJRMeAM = min(GknCxQFo) 
+| keep `BkQXJRMeAM`
+;
+
+BkQXJRMeAM:long
+4325287503714500302
+;
+
diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
index 4a45ddeeecba3..07f910c849c98 100644
--- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
+++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java
@@ -1091,7 +1091,13 @@ public enum Cap {
          * During resolution (pre-analysis) we have to consider that joins can override regex extracted values
          * see <a href="https://github.com/elastic/elasticsearch/issues/127467"> ES|QL: pruning of JOINs leads to missing fields #127467 </a>
          */
-        FIX_JOIN_MASKING_REGEX_EXTRACT;
+        FIX_JOIN_MASKING_REGEX_EXTRACT,
+
+        /**
+         * Avid GROK and DISSECT attributes being removed when resolving fields.
+         * see <a href="https://github.com/elastic/elasticsearch/issues/127468"> ES|QL: Grok only supports KEYWORD or TEXT values, found expression [type] type [INTEGER] #127468 </a>
+         */
+        KEEP_REGEX_EXTRACT_ATTRIBUTES;
 
         private final boolean enabled;
 
diff --git a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
index 4f0e06dc8d8f9..2e03e2123adab 100644
--- a/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
+++ b/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
@@ -656,7 +656,7 @@ static PreAnalysisResult fieldNames(LogicalPlan parsed, Set<String> enrichPolicy
             //
             // and ips_policy enriches the results with the same name ip field),
             // these aliases should be kept in the list of fields.
-            if (canRemoveAliases[0] && couldOverrideAliases(p)) {
+            if (canRemoveAliases[0] && p.anyMatch(EsqlSession::couldOverrideAliases)) {
                 canRemoveAliases[0] = false;
             }
             if (canRemoveAliases[0]) {
@@ -726,7 +726,8 @@ private static boolean couldOverrideAliases(LogicalPlan p) {
             || p instanceof Project
             || p instanceof RegexExtract
             || p instanceof Rename
-            || p instanceof TopN) == false;
+            || p instanceof TopN
+            || p instanceof UnresolvedRelation) == false;
     }
 
     private static boolean matchByName(Attribute attr, String other, boolean skipIfPattern) {
diff --git a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java
index 64f9404ca2a8a..f7d09eca2cce9 100644
--- a/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java
+++ b/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java
@@ -604,7 +604,20 @@ public void testEnrichEval() {
                 | eval language = concat(x, "-", lang)
                 | keep emp_no, x, lang, language
                 | sort emp_no desc | limit 3""",
-            Set.of("languages", "languages.*", "emp_no", "emp_no.*", "language_name", "language_name.*", "x", "x.*", "lang", "lang.*")
+            Set.of(
+                "emp_no",
+                "x",
+                "lang",
+                "language",
+                "language_name",
+                "languages",
+                "x.*",
+                "language_name.*",
+                "languages.*",
+                "emp_no.*",
+                "lang.*",
+                "language.*"
+            )
         );
     }
 
@@ -1355,7 +1368,7 @@ public void testAvoidGrokAttributesRemoval() {
             | grok type "%{WORD:b}"
             | stats x = max(b)
             | keep x""", Set.of());
-        assertThat(fieldNames, equalTo(Set.of("message", "x", "x.*", "message.*")));
+        assertThat(fieldNames, equalTo(Set.of("x", "b", "type", "message", "x.*", "message.*", "type.*", "b.*")));
     }
 
     public void testAvoidGrokAttributesRemoval2() {
@@ -1388,6 +1401,60 @@ public void testAvoidGrokAttributesRemoval3() {
 
     }
 
+    /**
+     * @see <a href="https://github.com/elastic/elasticsearch/issues/127468">ES|QL: Grok only supports KEYWORD or TEXT values, found expression [type] type [INTEGER]</a>
+     */
+    public void testAvoidGrokAttributesRemoval4() {
+        assumeTrue("LOOKUP JOIN available as snapshot only", EsqlCapabilities.Cap.JOIN_LOOKUP_V12.isEnabled());
+        Set<String> fieldNames = fieldNames("""
+            from message_types
+            | eval type = 1
+            | lookup join message_types_lookup on message
+            | drop  message
+            | grok type "%{WORD:b}"
+            | stats x = max(b)
+            | keep x""", Set.of());
+        assertThat(fieldNames, equalTo(Set.of("x", "b", "type", "message", "x.*", "message.*", "type.*", "b.*")));
+    }
+
+    /**
+     * @see <a href="https://github.com/elastic/elasticsearch/issues/127468">ES|QL: Grok only supports KEYWORD or TEXT values, found expression [type] type [INTEGER]</a>
+     */
+    public void testAvoidGrokAttributesRemoval5() {
+        assumeTrue("LOOKUP JOIN available as snapshot only", EsqlCapabilities.Cap.JOIN_LOOKUP_V12.isEnabled());
+        Set<String> fieldNames = fieldNames("""
+            FROM sample_data, employees
+            | EVAL client_ip = client_ip::keyword
+            | RENAME languages AS language_code
+            | LOOKUP JOIN clientips_lookup ON client_ip
+            | EVAL type = 1::keyword
+            | EVAL type = 2
+            | LOOKUP JOIN message_types_lookup ON message
+            | LOOKUP JOIN languages_lookup ON language_code
+            | DISSECT type "%{type_as_text}"
+            | KEEP message
+            | WHERE message IS NOT NULL
+            | SORT message DESC
+            | LIMIT 1""", Set.of());
+        assertThat(
+            fieldNames,
+            equalTo(
+                Set.of(
+                    "message",
+                    "type",
+                    "languages",
+                    "client_ip",
+                    "language_code",
+                    "language_code.*",
+                    "client_ip.*",
+                    "message.*",
+                    "type.*",
+                    "languages.*"
+                )
+            )
+        );
+    }
+
     public void testEnrichOnDefaultField() {
         Set<String> fieldNames = fieldNames("""
             from employees


Reviews:
user: login: astefan
body: @kanoshiou I really appreciate you provided this PR and you tried to understand where the fix needs to go.

Unfortunately, this is a trivial fix that doesn't quite catch the true reason for the initial failure.

The essence of the failure is the fact that `lookup join` can override the `type` user-provided field with its own `type` field. The `type` from `eval` is of type `integer` while the `type` that comes from the `lookup join` is of type `keyword`. Because the list of field names didn't contain `type` the Analyzer couldn't learn about that `type` that exists in `message_types_lookup` and the only thing it knew about its type is that it comes from `eval`.

`lookup join` (like `enrich` and maybe `inlinestats`) are special in the sense that they add some columns to the result, so some of the rules we have in `EsqlSession.fieldNames` need to be adjusted to account for these special characteristics. Because `lookup join` can add a field of the same name as `type` coming from an `eval` **before** the said lookup join command, we have [a special check in `fieldNames` that forbids the removal of those `Alias`es](https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java#L660-L674).

Your proposed solution does this:
- places the input references of grok in a new variable, instead of the `referenceBuilder` that was used before
- the special check above for lookup join is not applied to `referenceBuilder` for `grok` because the input reference is not there anymore
- you add to `referenceBuilder` the content of the grok-special variable

This is a bypass of the `lookup join` special removal check.

Instead, let the `grok` references in `referenceBuilder` where they are and don't let the special `lookup join` check remove it. The issue is with the `lookup join` check, not with `grok`, so adjust that one. I have the feeling that check is applicable in other cases, only that we didn't catch those yet.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T17:13:15Z
user: login: astefan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T09:13:41Z
user: login: astefan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T09:55:33Z
user: login: kanoshiou
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T10:03:59Z
user: login: astefan
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T11:32:58Z

Review comments:
diff_hunk: @@ -1341,6 +1341,18 @@ public void testDissectOverwriteName() {
         assertThat(fieldNames, equalTo(Set.of("emp_no", "emp_no.*", "first_name", "first_name.*")));
     }
 
+    public void testAvoidGrokAttributesRemoval() {
path: x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java
user: login: astefan
body: This one will need the lookup capability. Search for that one in tests here in this class and add it to this test as well.
created_at: 2025-05-09T09:12:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -331,3 +331,19 @@ ROW a="b c d x"| DISSECT a "%{b} %{} %{d} %{}";
 a:keyword  | b:keyword | d:keyword
 b c d x    | b         | d
 ;
+
+avoidAttributesRemoval
+required_capability: keep_regex_extract_attributes
+required_capability: join_lookup_v12
+from message_types 
+| eval type = 1 
+| lookup join message_types_lookup on message 
+| drop message 
+| dissect type "%{b}" 
+| stats x = max(b) 
+| keep x
+;
+
+x:keyword
+Success
+;
path: x-pack/plugin/esql/qa/testFixtures/src/main/resources/dissect.csv-spec
user: login: astefan
body: Add one more tests here, please. It's slightly more convoluted and uses multiple lookup join commands:

```
FROM sample_data, employees
| EVAL client_ip = client_ip::keyword
| RENAME languages AS language_code
| LOOKUP JOIN clientips_lookup ON client_ip
| EVAL type = 1::keyword
| EVAL type = 2
| LOOKUP JOIN message_types_lookup ON message
| LOOKUP JOIN languages_lookup ON language_code
| DISSECT type "%{type_as_text}"
| KEEP message
| WHERE message IS NOT NULL
| SORT message DESC
| LIMIT 1
```
created_at: 2025-05-09T09:12:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1341,6 +1341,18 @@ public void testDissectOverwriteName() {
         assertThat(fieldNames, equalTo(Set.of("emp_no", "emp_no.*", "first_name", "first_name.*")));
     }
 
path: x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/session/IndexResolverFieldNamesTests.java
user: login: astefan
body: Here add, also, the test that I mentioned for dissect.csv-spec:

```
FROM sample_data, employees
| EVAL client_ip = client_ip::keyword
| RENAME languages AS language_code
| LOOKUP JOIN clientips_lookup ON client_ip
| EVAL type = 1::keyword
| EVAL type = 2
| LOOKUP JOIN message_types_lookup ON message
| LOOKUP JOIN languages_lookup ON language_code
| DISSECT type "%{type_as_text}"
| KEEP message
| WHERE message IS NOT NULL
| SORT message DESC
| LIMIT 1
```
created_at: 2025-05-09T09:13:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -667,13 +667,22 @@ static PreAnalysisResult fieldNames(LogicalPlan parsed, Set<String> enrichPolicy
                 // remove the UnresolvedAttribute "x", since that is an Alias defined in "eval"
                 AttributeSet planRefs = p.references();
                 Set<String> fieldNames = planRefs.names();
-                p.forEachExpressionDown(Alias.class, alias -> {
-                    // do not remove the UnresolvedAttribute that has the same name as its alias, ie "rename id AS id"
-                    // or the UnresolvedAttributes that are used in Functions that have aliases "STATS id = MAX(id)"
-                    if (fieldNames.contains(alias.name())) {
-                        return;
+                canRemoveAliases[1] = true;
+                // go down each plan and remove aliases until we meet a plan that can override aliases
+                p.forEachDown(c -> {
path: x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
user: login: astefan
body: The "forEachDown" is already performed [here](https://github.com/elastic/elasticsearch/pull/127563/files#diff-40060e2ec9003953a228c4a03bdc80a301949b0d4e3dccc1978798e33a992a73R607) and I don't see the benefit here for the added nested loop.

Lets use my suggestion [here](https://github.com/elastic/elasticsearch/pull/127563#issuecomment-2865787666) combined with the UnresolvedRelation check in `couldOverrideAliases` [here](https://github.com/elastic/elasticsearch/pull/127563#issuecomment-2866223184). Let me know if you will make the change; otherwise I'm happy to contribute them to your PR together with the slightly modified tests. Thanks.
created_at: 2025-05-20T09:55:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -667,13 +667,22 @@ static PreAnalysisResult fieldNames(LogicalPlan parsed, Set<String> enrichPolicy
                 // remove the UnresolvedAttribute "x", since that is an Alias defined in "eval"
                 AttributeSet planRefs = p.references();
                 Set<String> fieldNames = planRefs.names();
-                p.forEachExpressionDown(Alias.class, alias -> {
-                    // do not remove the UnresolvedAttribute that has the same name as its alias, ie "rename id AS id"
-                    // or the UnresolvedAttributes that are used in Functions that have aliases "STATS id = MAX(id)"
-                    if (fieldNames.contains(alias.name())) {
-                        return;
+                canRemoveAliases[1] = true;
+                // go down each plan and remove aliases until we meet a plan that can override aliases
+                p.forEachDown(c -> {
path: x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java
user: login: kanoshiou
body: I'm happy to make the change, please give me a moment.
created_at: 2025-05-20T10:03:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: elasticsearchmachine
created_at: 2025-05-01T08:01:30Z
body: Pinging @elastic/es-analytical-engine (Team:Analytics)
author_association: COLLABORATOR
user: login: astefan
created_at: 2025-05-05T14:03:49Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-05T15:32:12Z
body: I'm really confused. I can get a result of `4325287503714500000` when I run the query on my local machine, but the test says it should be `4325287503714500302`. It seems the test result is right, I will update the branch.
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-08T16:28:03Z
body: @astefan Thank you for your review and for taking the time to provide such a detailed explanation of the issue's root cause. Your insights have been incredibly helpful, and I truly appreciate your guidance on my code, it has given me a much deeper understanding. I’ve learned a lot from your feedback!

I have reverted the bypass and implemented a modification that I believe addresses the issue. If there are any areas that need improvement or further adjustments, please let me know, I’d be happy to refine it further.
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-09T09:10:59Z
body: @kanoshiou I have a simpler suggestion for the change in EsqlSession. Instead of the original code in `main`

```
if (canRemoveAliases[0] && couldOverrideAliases(p)) {
```

we can do:

```
if (canRemoveAliases[0] && p.anyMatch(c -> couldOverrideAliases(c))) {
```

It will slightly change some of the existent tests in `IndexResolverFieldNamesTests` in terms of which `eval` generated field name will be put in the list of `fieldNames`, but I think that's acceptable.

Please, try it out and let me know your thoughts.
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-09T09:26:55Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-09T11:28:43Z
body: Thank you for your review and suggestions, @astefan!

I believe the statement `p.anyMatch(c -> couldOverrideAliases(c))` will always return true since there must be an `UnresolvedRelation` at the end. If we disregard the `UnresolvedRelation` case, I believe your code essentially avoids alias removal whenever there is a `lookup join` or `enrich` operation in the tree. However, this contradicts the original intent of minimizing the number of attributes required from `field_caps` at pre-analysis time.
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-09T11:38:16Z
body: > I believe the statement p.anyMatch(c -> couldOverrideAliases(c)) will always return true since there must be an UnresolvedRelation at the end

Indeed, I forgot about this part. UnresolvedRelation needs to also be accounted for in `couldOverrideAliases` method.

> I believe your code essentially avoids alias removal whenever there is a lookup join or enrich operation in the tree

Yes

> However, this contradicts the original intent of minimizing the number of attributes required from field_caps at pre-analysis time.

Yes, it's a fine balance. Being too aggressive in minimizing the list of fields could lead to some of the issues we've had recently. Having more fields in that list, but still avoiding to ask for all the fields (`*`) which is what we had before, is still an improvement.
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-09T12:56:13Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-09T13:04:42Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-12T03:03:43Z
body: @astefan The original code goes down each expression to remove aliases when the current plan passes the removal check. However, we cannot determine whether the aliases were defined before `lookup join`. 

To address this, my solution is:
- Go down each plan instead of each expression
- Check whether the plan can override aliases
  - If it can, stop removing further aliases
  - If it cannot, remove aliases and proceed to the next plan

I believe this solution is well-optimized and should prevent any future issues caused by alias removal check, since no more aliases before `lookup join` can execute this removal logic.
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-20T10:41:35Z
body: Thank you for your review @astefan! Your suggestion has been applied in [ee9de1c](https://github.com/elastic/elasticsearch/pull/127563/commits/ee9de1c6197dd15b953083430b00c704e634289c).
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-20T11:15:59Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-20T12:25:22Z
body: @kanoshiou please, look into the conflicts. Most likely, they are coming from the freshly merged PR.
author_association: CONTRIBUTOR
user: login: kanoshiou
created_at: 2025-05-20T12:27:50Z
body: @astefan conflicts resolved.
author_association: CONTRIBUTOR
user: login: astefan
created_at: 2025-05-20T12:29:51Z
body: buildkite test this
author_association: CONTRIBUTOR
user: login: elasticsearchmachine
created_at: 2025-05-21T07:20:54Z
body: ## 💚 Backport successful
| Status | Branch | Result |
|:------:|:------:|:------:|
| ✅ |  [8.19](https://github.com/elastic/elasticsearch/pull/128231)  | [<img src="https://img.shields.io/github/pulls/detail/state/elastic/elasticsearch/128231">](https://github.com/elastic/elasticsearch/pull/128231) |
| ✅ |  [9.0](https://github.com/elastic/elasticsearch/pull/128232)  | [<img src="https://img.shields.io/github/pulls/detail/state/elastic/elasticsearch/128232">](https://github.com/elastic/elasticsearch/pull/128232) |
author_association: COLLABORATOR
user: login: kanoshiou
created_at: 2025-05-21T07:20:55Z
body: Thank you @astefan!
author_association: CONTRIBUTOR