Repository: elastic/logstash
Pull request #17351: Improve the key validation in secret identifier.
Author: login: mashhurs
Labels: name: backport-active-9
description: Automated backport with mergify to all the active 9.[0-9]+ branches
name: backport-active-8
description: Automated backport with mergify to all the active 8.[0-9]+ branches
Created: 2025-03-19T06:12:31Z | Merged: 2025-04-24T04:14:18Z

## Release notes

 - The keystore has improved validation, and will no longer allow the creation of secrets that _cannot_ be accessed with config replacement parameters (`${` ... `}`). 

## What does this PR do?
Before this change, key-value can be added to keystore but it might impossible to reference the key because `ConfigVariableExpander` ignores if key name doesn't align with `SUBSTITUTION_PLACEHOLDER_REGEX`
Improves user experience with the secret store CLI that when providing key name, LS validates to accept a value which can be used in pipeline (aligns with `ConfigVariableExpander#SUBSTITUTION_PLACEHOLDER_REGEX`).
If already invalid key(s) were added, it warns to remove/replace the keys.

## Why is it important/What is the impact to the user?
No user impact. Keystore CLI restricts adding meaningless keys to keystore.

## Checklist

- [x] My code follows the style guidelines of this project
- [x] I have commented my code, particularly in hard-to-understand areas
- ~~[] I have made corresponding changes to the documentation~~
- ~~[] I have made corresponding change to the default configuration files (and/or docker env variables)~~
- [x] I have added tests that prove my fix is effective or that my feature works

## Author's Checklist

- [ ]

## How to test this PR locally

## Related issues

- 

## Use cases


## Screenshots


## Logs




Diff:
diff --git a/docs/reference/keystore.md b/docs/reference/keystore.md
index 9aae6fd7482..1d57209d8c6 100644
--- a/docs/reference/keystore.md
+++ b/docs/reference/keystore.md
@@ -136,7 +136,7 @@ bin/logstash-keystore add ES_USER ES_PWD
 When prompted, enter a value for each key.
 
 ::::{note}
-Key values are limited to ASCII characters. It includes digits, letters, and a few special symbols.
+Key values are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), underscores (`_`), and dots (`.`); they must be at least one character long and cannot begin with a number.
 ::::
 
 
diff --git a/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java b/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
index 008ddc599d6..def0a7a1620 100644
--- a/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
+++ b/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
@@ -35,11 +35,14 @@
  * */
 public class ConfigVariableExpander implements AutoCloseable {
 
-    private static String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>[a-zA-Z_.][a-zA-Z0-9_.]*)(:(?<default>[^}]*))?}";
-
-    private Pattern substitutionPattern = Pattern.compile(SUBSTITUTION_PLACEHOLDER_REGEX);
-    private SecretStore secretStore;
-    private EnvironmentVariableProvider envVarProvider;
+    public static final Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
+    public static final String KEY_PATTERN_DESCRIPTION = "Key names are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), " +
+            "underscores (`_`), and dots (`.`); they must be at least one character long and cannot begin with a number";
+    private static final String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>" + KEY_PATTERN + ")(:(?<default>[^}]*))?}";
+
+    private static final Pattern substitutionPattern = Pattern.compile(SUBSTITUTION_PLACEHOLDER_REGEX);
+    private final SecretStore secretStore;
+    private final EnvironmentVariableProvider envVarProvider;
 
     /**
      * Creates a ConfigVariableExpander that doesn't lookup any secreted placeholder.
diff --git a/logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java b/logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
index 08f43abb1ee..2e5bf353e12 100644
--- a/logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
+++ b/logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
@@ -20,6 +20,11 @@
 
 package org.logstash.secret;
 
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.util.Strings;
+import org.logstash.plugins.ConfigVariableExpander;
+
 import java.util.Locale;
 import java.util.regex.Pattern;
 
@@ -34,12 +39,14 @@ public class SecretIdentifier {
     private final static Pattern urnPattern = Pattern.compile("urn:logstash:secret:"+ VERSION + ":.*$");
     private final String key;
 
+    private static final Logger logger = LogManager.getLogger(SecretIdentifier.class);
+
     /**
      * Constructor
      *
      * @param key The unique part of the identifier. This is the key to reference the secret, and the key itself should not be sensitive. For example: {@code db.pass}
      */
-    public SecretIdentifier(String key) {
+    public SecretIdentifier(final String key) {
         this.key = validateWithTransform(key, "key");
     }
 
@@ -49,7 +56,7 @@ public SecretIdentifier(String key) {
      * @param urn The {@link String} formatted identifier obtained originally from {@link SecretIdentifier#toExternalForm()}
      * @return The {@link SecretIdentifier} object used to identify secrets, null if not valid external form.
      */
-    public static SecretIdentifier fromExternalForm(String urn) {
+    public static SecretIdentifier fromExternalForm(final String urn) {
         if (urn == null || !urnPattern.matcher(urn).matches()) {
             throw new IllegalArgumentException("Invalid external form " + urn);
         }
@@ -57,20 +64,6 @@ public static SecretIdentifier fromExternalForm(String urn) {
         return new SecretIdentifier(validateWithTransform(parts[4], "key"));
     }
 
-    /**
-     * Minor validation and downcases the parts
-     *
-     * @param part     The part of the URN to validate
-     * @param partName The name of the part used for logging.
-     * @return The validated and transformed part.
-     */
-    private static String validateWithTransform(String part, String partName) {
-        if (part == null || part.isEmpty()) {
-            throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
-        }
-        return part.toLowerCase(Locale.US);
-    }
-
     @Override
     public boolean equals(Object o) {
         if (this == o) return true;
@@ -118,4 +111,22 @@ public String toExternalForm() {
     public String toString() {
         return toExternalForm();
     }
+
+    /**
+     * Minor validation and downcases the parts
+     *
+     * @param key      The key, a part of the URN to validate.
+     * @param partName The name of the part used for logging.
+     * @return The validated and transformed part.
+     */
+    private static String validateWithTransform(final String key, final String partName) {
+        if (key == null || key.isEmpty() || Strings.isBlank(key)) {
+            throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
+        }
+
+        if (!ConfigVariableExpander.KEY_PATTERN.matcher(key).matches()) {
+            logger.warn(String.format("Invalid secret key name `%s` provided. %s", key, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
+        }
+        return key.toLowerCase(Locale.US);
+    }
 }
diff --git a/logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java b/logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java
index 0cf277aa03a..f83fac87e04 100644
--- a/logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java
+++ b/logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java
@@ -20,13 +20,23 @@
 
 package org.logstash.secret.cli;
 
+import org.logstash.plugins.ConfigVariableExpander;
 import org.logstash.secret.SecretIdentifier;
-import org.logstash.secret.store.*;
+import org.logstash.secret.store.SecretStore;
+import org.logstash.secret.store.SecretStoreFactory;
+import org.logstash.secret.store.SecretStoreUtil;
+import org.logstash.secret.store.SecureConfig;
 
 import java.nio.CharBuffer;
 import java.nio.charset.CharsetEncoder;
 import java.nio.charset.StandardCharsets;
-import java.util.*;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
 import java.util.stream.Collectors;
 
 import static org.logstash.secret.store.SecretStoreFactory.LOGSTASH_MARKER;
@@ -178,7 +188,7 @@ Set<CommandOptions> getValidOptions() {
         }
     }
 
-    public SecretStoreCli(Terminal terminal){
+    public SecretStoreCli(Terminal terminal) {
         this(terminal, SecretStoreFactory.fromEnvironment());
     }
 
@@ -189,9 +199,10 @@ public SecretStoreCli(Terminal terminal){
 
     /**
      * Entry point to issue a command line command.
+     *
      * @param primaryCommand The string representation of a {@link SecretStoreCli.Command}, if the String does not map to a {@link SecretStoreCli.Command}, then it will show the help menu.
-     * @param config The configuration needed to work a secret store. May be null for help.
-     * @param allArguments This can be either identifiers for a secret, or a sub command like --help. May be null.
+     * @param config         The configuration needed to work a secret store. May be null for help.
+     * @param allArguments   This can be either identifiers for a secret, or a sub command like --help. May be null.
      */
     public void command(String primaryCommand, SecureConfig config, String... allArguments) {
         terminal.writeLine("");
@@ -212,7 +223,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
         final CommandLine commandLine = commandParseResult.get();
         switch (commandLine.getCommand()) {
             case CREATE: {
-                if (commandLine.hasOption(CommandOptions.HELP)){
+                if (commandLine.hasOption(CommandOptions.HELP)) {
                     terminal.writeLine("Creates a new keystore. For example: 'bin/logstash-keystore create'");
                     return;
                 }
@@ -227,7 +238,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
                 break;
             }
             case LIST: {
-                if (commandLine.hasOption(CommandOptions.HELP)){
+                if (commandLine.hasOption(CommandOptions.HELP)) {
                     terminal.writeLine("List all secret identifiers from the keystore. For example: " +
                             "`bin/logstash-keystore list`. Note - only the identifiers will be listed, not the secrets.");
                     return;
@@ -239,7 +250,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
                 break;
             }
             case ADD: {
-                if (commandLine.hasOption(CommandOptions.HELP)){
+                if (commandLine.hasOption(CommandOptions.HELP)) {
                     terminal.writeLine("Add secrets to the keystore. For example: " +
                             "`bin/logstash-keystore add my-secret`, at the prompt enter your secret. You will use the identifier ${my-secret} in your Logstash configuration.");
                     return;
@@ -251,6 +262,9 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
                 if (secretStoreFactory.exists(config.clone())) {
                     final SecretStore secretStore = secretStoreFactory.load(config);
                     for (String argument : commandLine.getArguments()) {
+                        if (!ConfigVariableExpander.KEY_PATTERN.matcher(argument).matches()) {
+                            throw new IllegalArgumentException(String.format("Invalid secret key name `%s` provided. %s", argument, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
+                        }
                         final SecretIdentifier id = new SecretIdentifier(argument);
                         final byte[] existingValue = secretStore.retrieveSecret(id);
                         if (existingValue != null) {
@@ -263,7 +277,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
 
                         final String enterValueMessage = String.format("Enter value for %s: ", argument);
                         char[] secret = null;
-                        while(secret == null) {
+                        while (secret == null) {
                             terminal.write(enterValueMessage);
                             final char[] readSecret = terminal.readSecret();
 
@@ -288,7 +302,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
                 break;
             }
             case REMOVE: {
-                if (commandLine.hasOption(CommandOptions.HELP)){
+                if (commandLine.hasOption(CommandOptions.HELP)) {
                     terminal.writeLine("Remove secrets from the keystore. For example: " +
                             "`bin/logstash-keystore remove my-secret`");
                     return;
@@ -314,7 +328,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
         }
     }
 
-    private void printHelp(){
+    private void printHelp() {
         terminal.writeLine("Usage:");
         terminal.writeLine("--------");
         terminal.writeLine("bin/logstash-keystore [option] command [argument]");
diff --git a/logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java b/logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java
index 7416a7ec988..2aee8e13832 100644
--- a/logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java
+++ b/logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java
@@ -35,7 +35,7 @@
 import java.util.Map;
 import java.util.Optional;
 import java.util.Queue;
-import java.util.UUID;
+import java.util.Random;
 
 import static junit.framework.TestCase.assertTrue;
 import static org.assertj.core.api.Assertions.assertThat;
@@ -147,8 +147,8 @@ public void testAddEmptyValue() {
         terminal.in.add(""); // sets the empty value
         terminal.in.add("value");
 
-        String id = UUID.randomUUID().toString();
-        cli.command("add", newStoreConfig.clone(), id);
+        final String keyName = getRandomKeyName();
+        cli.command("add", newStoreConfig.clone(), keyName);
         assertThat(terminal.out).containsIgnoringCase("ERROR: Value cannot be empty");
     }
 
@@ -159,7 +159,7 @@ public void testAddNonAsciiValue() {
         terminal.in.add("€€€€€"); // sets non-ascii value value
         terminal.in.add("value");
 
-        String id = UUID.randomUUID().toString();
+        final String id = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), id);
         assertThat(terminal.out).containsIgnoringCase("ERROR: Value must contain only ASCII characters");
     }
@@ -168,8 +168,8 @@ public void testAddNonAsciiValue() {
     public void testAdd() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString()); // sets the value
-        String id = UUID.randomUUID().toString();
+        terminal.in.add(getRandomKeyName()); // sets the value
+        final String id = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), id);
         terminal.reset();
 
@@ -197,11 +197,11 @@ public void testAddWithNoIdentifiers() {
     public void testAddMultipleKeys() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString());
-        terminal.in.add(UUID.randomUUID().toString());
+        terminal.in.add(getRandomKeyName());
+        terminal.in.add(getRandomKeyName());
 
-        final String keyOne = UUID.randomUUID().toString();
-        final String keyTwo = UUID.randomUUID().toString();
+        final String keyOne = getRandomKeyName();
+        final String keyTwo = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), keyOne, keyTwo);
         terminal.reset();
 
@@ -211,7 +211,7 @@ public void testAddMultipleKeys() {
 
     @Test
     public void testAddWithoutCreatedKeystore() {
-        cli.command("add", newStoreConfig.clone(), UUID.randomUUID().toString());
+        cli.command("add", newStoreConfig.clone(), getRandomKeyName());
         assertThat(terminal.out).containsIgnoringCase("ERROR: Logstash keystore not found. Use 'create' command to create one.");
     }
 
@@ -219,10 +219,10 @@ public void testAddWithoutCreatedKeystore() {
     public void testAddWithStdinOption() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString()); // sets the value
-        terminal.in.add(UUID.randomUUID().toString()); // sets the value
+        terminal.in.add(getRandomKeyName()); // sets the value
+        terminal.in.add(getRandomKeyName()); // sets the value
 
-        String id = UUID.randomUUID().toString();
+        final String id = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), id, SecretStoreCli.CommandOptions.STDIN.getOption());
         terminal.reset();
 
@@ -235,8 +235,8 @@ public void testAddWithStdinOption() {
     public void testRemove() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString()); // sets the value
-        String id = UUID.randomUUID().toString();
+        terminal.in.add(getRandomKeyName()); // sets the value
+        final String id = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), id);
         System.out.println(terminal.out);
         terminal.reset();
@@ -256,11 +256,11 @@ public void testRemove() {
     public void testRemoveMultipleKeys() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString());
-        terminal.in.add(UUID.randomUUID().toString());
+        terminal.in.add(getRandomKeyName());
+        terminal.in.add(getRandomKeyName());
 
-        final String keyOne = UUID.randomUUID().toString();
-        final String keyTwo = UUID.randomUUID().toString();
+        final String keyOne = getRandomKeyName();
+        final String keyTwo = getRandomKeyName();
 
         cli.command("add", newStoreConfig.clone(), keyOne, keyTwo);
         terminal.reset();
@@ -281,8 +281,8 @@ public void testRemoveMultipleKeys() {
     public void testRemoveMissing() {
         createKeyStore();
 
-        terminal.in.add(UUID.randomUUID().toString()); // sets the value
-        String id = UUID.randomUUID().toString();
+        terminal.in.add(getRandomKeyName()); // sets the value
+        final String id = getRandomKeyName();
         cli.command("add", newStoreConfig.clone(), id);
         System.out.println(terminal.out);
         terminal.reset();
@@ -318,7 +318,7 @@ public void testCommandWithUnrecognizedOption() {
         terminal.in.add("foo");
 
         final String invalidOption = "--invalid-option";
-        cli.command("add", newStoreConfig.clone(), UUID.randomUUID().toString(), invalidOption);
+        cli.command("add", newStoreConfig.clone(), getRandomKeyName(), invalidOption);
         assertThat(terminal.out).contains(String.format("Unrecognized option '%s' for command 'add'", invalidOption));
 
         terminal.reset();
@@ -387,6 +387,23 @@ private void assertNotListed(String... expected) {
         assertTrue(Arrays.stream(expected).noneMatch(terminal.out::contains));
     }
 
+    private final static String KEY_FIRST_CHAR_CANDIDATES = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_.";
+    private final static String KEY_REST_CHARS = KEY_FIRST_CHAR_CANDIDATES.concat("01234567890");
+    private String getRandomKeyName() {
+        final Random random = new Random();
+        final StringBuilder sb = new StringBuilder();
+
+        // add a random legal first-character
+        sb.append(KEY_FIRST_CHAR_CANDIDATES.charAt(random.nextInt(KEY_FIRST_CHAR_CANDIDATES.length())));
+
+        // add between 0 and 40 random legal rest-characters
+        final int more = 40;
+        random.ints(more, 0, KEY_REST_CHARS.length())
+                .mapToObj(KEY_REST_CHARS::charAt)
+                .forEach(sb::append);
+        return sb.toString().toLowerCase();
+    }
+
     private void assertPrimaryHelped() {
         assertThat(terminal.out).
                 containsIgnoringCase("Commands").


Reviews:
user: login: yaauie
body: Mechanically, this works, and safely guards against adding keys that cannot be referenced in the pipelines without breaking users whose keystores include offending keys.

I've left suggestions about DRY-ing it up a bit, and improving the admitedly-difficult description of the effective key pattern requirement.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T18:58:14Z
user: login: mashhurs
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T18:22:40Z
user: login: yaauie
body: 👍🏼 looking on-track.

I've left a comment about message format, and suggested more useful random key generation for tests.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T15:50:22Z
user: login: mashhurs
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T01:25:10Z
user: login: mashhurs
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T18:03:33Z
user: login: yaauie
body: Works as advertised.

I've left one nit-pick about making the validation step more cohesive, but you're free to decide whether to resolve it or not.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-23T22:26:22Z
user: login: mashhurs
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T04:13:30Z

Review comments:
diff_hunk: @@ -29,17 +33,22 @@
  */
 public class SecretIdentifier {
 
+    // aligns with ConfigVariableExpander substitution pattern
+    public final static Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: yaauie
body: Nit: referencing `ConfigVariableExpander` in a comment feels fragile.

I'd prefer the definition of the pattern to live in `ConfigVariableExpander`, exposed here as a static method akin to:

~~~ java
    public static void validateKey(final String candidate) throws InvalidKeyException {
        if(!ConfigVariableExpander.KEY_PATTERN.matcher(candidate).matches()) {
            throw new InvalidKeyException(String.format("Invalid secret key name `%s` provided.", candidate) + ConfigVariableExpander.KEY_PATTERN_DESCRIPTION);
        }
    }
~~~

where:
 - `InvalidKeyException` is a new subclass of `SecretStoreException`
 - `ConfigVariableExpander` exposes `KEY_PATTERN` and `KEY_PATTERN_DESCRIPTION`:
   > ~~~ diff
   > diff --git a/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java b/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
   > index 008ddc599..dc9e0d51d 100644
   > --- a/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
   > +++ b/logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java
   > @@ -35,7 +35,10 @@ import java.util.regex.Pattern;
   >   * */
   >  public class ConfigVariableExpander implements AutoCloseable {
   >  
   > -    private static String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>[a-zA-Z_.][a-zA-Z0-9_.]*)(:(?<default>[^}]*))?}";
   > +    public static Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
   > +    public static String KEY_PATTERN_DESCRIPTION = "Key names are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), underscores (`_`), and dots (`.`); " +
   > +            "they must be at least one character long and cannot begin with a number.";
   > +    private static String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>" + KEY_PATTERN + ")(:(?<default>[^}]*))?}";
   >  
   >      private Pattern substitutionPattern = Pattern.compile(SUBSTITUTION_PLACEHOLDER_REGEX);
   >      private SecretStore secretStore;
   > ~~~




created_at: 2025-04-08T17:49:13Z
author_association: MEMBER
diff_hunk: @@ -136,7 +136,7 @@ bin/logstash-keystore add ES_USER ES_PWD
 When prompted, enter a value for each key.
 
 ::::{note}
-Key values are limited to ASCII characters. It includes digits, letters, and a few special symbols.
+Key values are limited to ASCII characters. It can only accept digits, letters and optional `.` (dots) and/or `_` (underscores) symbols included between or after.
path: docs/reference/keystore.md
user: login: yaauie
body: Describing patterns in plain language is _hard_.

The pattern we are describing is `[a-zA-Z_.][a-zA-Z0-9_.]*`, which I find easiest to describe like:

```suggestion
Key values are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), underscores (`_`), and dots (`.`); they must be at least one character long and cannot begin with a number.
```

_Technically, we support patterns like `.` or `._.` which are rather silly names for keys, but refining what we allow is out of scope for this PR (and would be a breaking change, complete with a deprecation window to allow users to migrate to supported names before enforcing a new requirement)_.
created_at: 2025-04-08T17:54:37Z
author_association: MEMBER
diff_hunk: @@ -60,15 +69,19 @@ public static SecretIdentifier fromExternalForm(String urn) {
     /**
      * Minor validation and downcases the parts
      *
-     * @param part     The part of the URN to validate
+     * @param key      The key, a part of the URN to validate.
      * @param partName The name of the part used for logging.
      * @return The validated and transformed part.
      */
-    private static String validateWithTransform(String part, String partName) {
-        if (part == null || part.isEmpty()) {
+    private static String validateWithTransform(final String key, final String partName) {
+        if (key == null || key.isEmpty() || Strings.isBlank(key)) {
             throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
         }
-        return part.toLowerCase(Locale.US);
+
+        if (!KEY_PATTERN.matcher(key).matches()) {
+            logger.warn(String.format("Invalid %s key appeared in keystore. Please remove it as it cannot be used in the pipelines", key));
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: yaauie
body: nit: backtick-quote the user input

```suggestion
            logger.warn(String.format("Invalid `%s` key appeared in keystore. Please remove it as it cannot be used in the pipelines", key));
```
created_at: 2025-04-08T18:38:59Z
author_association: MEMBER
diff_hunk: @@ -60,15 +69,19 @@ public static SecretIdentifier fromExternalForm(String urn) {
     /**
      * Minor validation and downcases the parts
      *
-     * @param part     The part of the URN to validate
+     * @param key      The key, a part of the URN to validate.
      * @param partName The name of the part used for logging.
      * @return The validated and transformed part.
      */
-    private static String validateWithTransform(String part, String partName) {
-        if (part == null || part.isEmpty()) {
+    private static String validateWithTransform(final String key, final String partName) {
+        if (key == null || key.isEmpty() || Strings.isBlank(key)) {
             throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
         }
-        return part.toLowerCase(Locale.US);
+
+        if (!KEY_PATTERN.matcher(key).matches()) {
+            logger.warn(String.format("Invalid %s key appeared in keystore. Please remove it as it cannot be used in the pipelines", key));
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: yaauie
body: If we take my other suggestion to have the pattern and its description available from `ConfigVariableExpander`, this will also benefit from including the description of the pattern.
created_at: 2025-04-08T18:56:58Z
author_association: MEMBER
diff_hunk: @@ -29,17 +33,22 @@
  */
 public class SecretIdentifier {
 
+    // aligns with ConfigVariableExpander substitution pattern
+    public final static Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: mashhurs
body: Agree with moving the pattern to a `ConfigVariableExpander`.
In case of **_throwing exception_** when key doesn't match the pattern (in `validateKey()`), this _**would be breaking change**_. 
This would be a good candidate in major releases or if we still can, I can follow up (with separate task) with instructions to replace invalid keys.

created_at: 2025-04-09T18:22:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -251,6 +262,9 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
                 if (secretStoreFactory.exists(config.clone())) {
                     final SecretStore secretStore = secretStoreFactory.load(config);
                     for (String argument : commandLine.getArguments()) {
+                        if (!ConfigVariableExpander.KEY_PATTERN.matcher(argument).matches()) {
+                            throw new IllegalArgumentException(String.format("Invalid secret key name `%s` provided.", argument) + ConfigVariableExpander.KEY_PATTERN_DESCRIPTION);
path: logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java
user: login: yaauie
body: nit: I think I might have even done it in my prior review, but reading it here the mix of format strings and concatenation (~`String.format(template, var) + CONSTANT`) just _feels_ a bit weird to me; can we make room for it in the template?


```suggestion
                            throw new IllegalArgumentException(String.format("Invalid secret key name `%s` provided. %s", argument, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION);
```


If not, we're missing a space after the period:

```suggestion
                            throw new IllegalArgumentException(String.format("Invalid secret key name `%s` provided. ", argument) + ConfigVariableExpander.KEY_PATTERN_DESCRIPTION);
```
created_at: 2025-04-10T15:03:46Z
author_association: MEMBER
diff_hunk: @@ -387,6 +387,11 @@ private void assertNotListed(String... expected) {
         assertTrue(Arrays.stream(expected).noneMatch(terminal.out::contains));
     }
 
+    private String getRandomKeyName() {
+        final String[] uuidParts = UUID.randomUUID().toString().split("-");
+        return "r".concat(uuidParts[0]).concat("_").concat(uuidParts[1]);
path: logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java
user: login: yaauie
body: If were using UUID, it's easier to just replace the unsupported dashes ('-') with acceptable underscores, than to split and concat arbitrarily.

```suggestion
        return "r" + UUID.randomUUID().toString().replace('-','_');
```

Alternatively, we can generate the actual random keys that should be acceptable, which has the benefit of corroborating the requirement:

```suggestion
        final String firstCharacters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_.";
        final String restCharacters = firstCharacters + "01234567890";

        final Random random = new Random();
        final StringBuilder sb = new StringBuilder();

        // add a random legal first-character
        sb.append(firstCharacters.charAt(random.nextInt(firstCharacters.length())));

        // add between 0 and 40 random legal rest-characters
        final int more = random.nextInt(40);
        random.ints(more, 0, restCharacters.length())
                .mapToObj(restCharacters::charAt)
                .forEach(sb::append);
        
        return sb.toString();
```

created_at: 2025-04-10T15:40:36Z
author_association: MEMBER
diff_hunk: @@ -29,17 +33,22 @@
  */
 public class SecretIdentifier {
 
+    // aligns with ConfigVariableExpander substitution pattern
+    public final static Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: yaauie
body: d'oh. I had forgotten about the _warn_ use-case and was only addressing the error one. The way you resolved it makes sense.

If we don't _need_ to break users existing keystores with unreachable keys, I don't see value in introducing a breaking change. This PR already prevents them from creating new unreachable keys and warns when listing unreachable ones.
created_at: 2025-04-10T15:44:00Z
author_association: MEMBER
diff_hunk: @@ -118,4 +111,31 @@ public String toExternalForm() {
     public String toString() {
         return toExternalForm();
     }
+
+    /**
+     * Validates the provided key against null, empty and {@link ConfigVariableExpander#KEY_PATTERN}
+     * @param key a key to be validated
+     * @param keyName a key name
+     */
+    private static void validateKey(final String key, final String keyName) {
+        if (key == null || key.isEmpty() || Strings.isBlank(key)) {
+            throw new IllegalArgumentException(String.format("%s may not be null or empty", keyName));
+        }
+
+        if (!ConfigVariableExpander.KEY_PATTERN.matcher(key).matches()) {
+            logger.warn(String.format("Invalid secret key name `%s` provided.", key) + ConfigVariableExpander.KEY_PATTERN_DESCRIPTION);
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: mashhurs
body: ```suggestion
            logger.warn(String.format("Invalid secret key name `%s` provided. %s", key, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
```
created_at: 2025-04-11T01:23:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -387,6 +387,11 @@ private void assertNotListed(String... expected) {
         assertTrue(Arrays.stream(expected).noneMatch(terminal.out::contains));
     }
 
+    private String getRandomKeyName() {
+        final String[] uuidParts = UUID.randomUUID().toString().split("-");
+        return "r".concat(uuidParts[0]).concat("_").concat(uuidParts[1]);
path: logstash-core/src/test/java/org/logstash/secret/cli/SecretStoreCliTest.java
user: login: mashhurs
body: :) the funny thing is I initially did same and removed as it was getting longer lines.
This requires removing UUID import and adding Random. Also, when CLI lower cases the key, so we need to also lower case the random key name here.
I will verify on my local and push with your co-author.
created_at: 2025-04-11T18:03:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -118,4 +111,31 @@ public String toExternalForm() {
     public String toString() {
         return toExternalForm();
     }
+
+    /**
+     * Validates the provided key against null, empty and {@link ConfigVariableExpander#KEY_PATTERN}
+     * @param key a key to be validated
+     * @param keyName a key name
+     */
+    private static void validateKey(final String key, final String keyName) {
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: yaauie
body: nit: this new function is only called from `SecretIdentifier#validateWithTransform`, adding indirection while only functionally adding one more validation clause`.

If we were to reabsorb that clause into the previous `SecretIdentifier#validateWithTransform` it would look something like:

~~~ diff
     /**
      * Minor validation and downcases the parts
      *
      * @param part     The part of the URN to validate
      * @param partName The name of the part used for logging.
      * @return The validated and transformed part.
      */
     private static String validateWithTransform(String part, String partName) {
-        if (part == null || part.isEmpty()) {
+        if (part == null || part.isEmpty() || Strings.isBlank(part)) {
             throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
         }
+        if (!ConfigVariableExpander.KEY_PATTERN.matcher(part).matches()) {
+            logger.warn(String.format("Invalid secret %s name `%s` provided. %s", partName, part, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
+        }
         return part.toLowerCase(Locale.US);
     }
~~~


created_at: 2025-04-23T22:24:36Z
author_association: MEMBER
diff_hunk: @@ -118,4 +111,31 @@ public String toExternalForm() {
     public String toString() {
         return toExternalForm();
     }
+
+    /**
+     * Validates the provided key against null, empty and {@link ConfigVariableExpander#KEY_PATTERN}
+     * @param key a key to be validated
+     * @param keyName a key name
+     */
+    private static void validateKey(final String key, final String keyName) {
path: logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java
user: login: mashhurs
body: Yup makes sense, applied with [`700e596`](https://github.com/elastic/logstash/commit/700e596bc668fa26aec4b9db23a4ef6dea9029e9) commit.
created_at: 2025-04-24T04:13:30Z
author_association: CONTRIBUTOR

PR comments:
user: login: mergify[bot]
created_at: 2025-03-19T06:13:05Z
body: This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:
* `backport-8./d` is the label to automatically backport to the `8./d` branch. `/d` is the digit.
* `backport-8.x` is the label to automatically backport to the `8.x` branch.
* If no backport is necessary, please add the `backport-skip` label
author_association: CONTRIBUTOR
user: login: yaauie
created_at: 2025-03-19T22:03:51Z
body: > No users should be impacted unless they are using restricted symbols (["?", "..", "/", "\\", "'", "\"", "$", "*", "|", "<", ">", " "]) in the key.

This sounds like a breaking change. 

 - What was the prior experience when using a key that includes now-restricted characters?
 - What determined that these specific characters are restricted (while allowing a bunch of non-printing characters in the lower-ascii plane and the gamut of unicode characters above the lower-ascii plane)?
author_association: MEMBER
user: login: github-actions[bot]
created_at: 2025-03-22T03:20:20Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: mashhurs
created_at: 2025-03-23T03:42:56Z
body: > > No users should be impacted unless they are using restricted symbols (["?", "..", "/", "\", "'", """, "$", "*", "|", "<", ">", " "]) in the key.
> 
> This sounds like a breaking change.
> 
> * What was the prior experience when using a key that includes now-restricted characters?
> * What determined that these specific characters are restricted (while allowing a bunch of non-printing characters in the lower-ascii plane and the gamut of unicode characters above the lower-ascii plane)?

Yeah, didn't think of the case if keystore had invalid keys already included.
[`a3372c5`](https://github.com/elastic/logstash/pull/17351/commits/a3372c51d708f2374876a62cdf2eba1605bae6c6) commit changes the approach:
- to restrict adding invalid keys to keystore with CLI. Invalid means here outside of `ConfigVariableExpander#SUBSTITUTION_PLACEHOLDER_REGEX `;
- warns/guides users if their keystore already contains invalid keys;
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-09T16:38:58Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-09T17:30:13Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-09T18:16:56Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: mashhurs
created_at: 2025-04-09T20:33:45Z
body: Rebased against main as we were having CI issues, fixed by https://github.com/elastic/logstash/pull/17531
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-09T20:37:05Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-11T01:29:11Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-04-11T18:08:43Z
body: :page_with_curl: **DOCS PREVIEW** :sparkles: https://logstash_bk_17351.docs-preview.app.elstc.co/diff
author_association: CONTRIBUTOR
user: login: elastic-sonarqube[bot]
created_at: 2025-04-24T00:23:47Z
body: ## [![Quality Gate passed](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonar.elastic.dev/dashboard?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351) **Quality Gate passed**  
Issues  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&resolved=false) [0 New issues](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&resolved=false)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/fixed-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&fixedInPullRequest=17351) [0 Fixed issues](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&fixedInPullRequest=17351)  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/accepted-16px.png '')](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&issueStatuses=ACCEPTED) [0 Accepted issues](https://sonar.elastic.dev/project/issues?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&issueStatuses=ACCEPTED)

Measures  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/security_hotspots?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351) [0 Security Hotspots](https://sonar.elastic.dev/security_hotspots?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351)  
![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/no-data-16px.png '') No data about Coverage  
[![](https://raw.githubusercontent.com/SonarSource/sonarqube-static-resources/master/v104/common/passed-16px.png '')](https://sonar.elastic.dev/component_measures?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&metric=new_duplicated_lines_density&view=list) [0.0% Duplication on New Code](https://sonar.elastic.dev/component_measures?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube](https://sonar.elastic.dev/dashboard?id=elastic_logstash_AYm_nEbQaV3I-igkX1q9&pullRequest=17351)


author_association: NONE
user: login: elasticmachine
created_at: 2025-04-24T00:38:24Z
body: ## :green_heart: Build Succeeded
* [Buildkite Build](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2872)
* Commit: 700e596bc668fa26aec4b9db23a4ef6dea9029e9

### History
* :green_heart: [Build #2845](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2845) succeeded 6e816bc94fc7a52759b37fd5eb4d61a3ca38dd32
* :broken_heart: [Build #2840](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2840) failed 8e0136a7365514618ea1e2ab772ea61b0e5a8f45
* :green_heart: [Build #2823](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2823) succeeded bfea29d5c0e086114821d2bc1e3f239ac96c9008
* :broken_heart: [Build #2820](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2820) failed 6ac79ce9979c804d7a99d5ace54ee1944299415a
* :broken_heart: [Build #2819](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2819) failed 9139460c2438b94a636a4ac12dfc5164429a493b
* :broken_heart: [Build #2818](https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2818) failed 0d9c5af0d2cdf7f403625389297c790566c46939

cc @mashhurs

<!--buildkite-pr-comment-logstash-pull-request-pipeline
{"builds":[{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2872","number":2872,"commit":"700e596bc668fa26aec4b9db23a4ef6dea9029e9"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2845","number":2845,"commit":"6e816bc94fc7a52759b37fd5eb4d61a3ca38dd32"},{"buildStatus":{"state":"failed","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2840","number":2840,"commit":"8e0136a7365514618ea1e2ab772ea61b0e5a8f45"},{"buildStatus":{"state":"passed","success":true,"hasRetries":false,"hasNonPreemptionRetries":false},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2823","number":2823,"commit":"bfea29d5c0e086114821d2bc1e3f239ac96c9008"},{"buildStatus":{"state":"failed","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2820","number":2820,"commit":"6ac79ce9979c804d7a99d5ace54ee1944299415a"},{"buildStatus":{"state":"failing","success":false,"hasRetries":true,"hasNonPreemptionRetries":true},"url":"https://buildkite.com/elastic/logstash-pull-request-pipeline/builds/2819","number":2819,"commit":"9139460c2438b94a636a4ac12dfc5164429a493b"}],"number":2872}
buildkite-pr-comment-->
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-24T04:14:31Z
body: @mergifyio backport 8.17 8.18 8.19 9.0
author_association: CONTRIBUTOR
user: login: mergify[bot]
created_at: 2025-04-24T04:14:56Z
body: > backport 8.17 8.18 8.19 9.0

#### ✅ Backports have been created

<details>

* [#17585 [8.17] (backport #17351) Improve the key validation in secret identifier.](https://github.com/elastic/logstash/pull/17585) has been created for branch `8.17` but encountered conflicts
* [#17586 [8.18] (backport #17351) Improve the key validation in secret identifier.](https://github.com/elastic/logstash/pull/17586) has been created for branch `8.18` but encountered conflicts
* [#17587 [8.19] (backport #17351) Improve the key validation in secret identifier.](https://github.com/elastic/logstash/pull/17587) has been created for branch `8.19` but encountered conflicts
* [#17588 [9.0] (backport #17351) Improve the key validation in secret identifier.](https://github.com/elastic/logstash/pull/17588) has been created for branch `9.0`

</details>


<!---
DO NOT EDIT
-*- Mergify Payload -*-
{"command": "backport 8.17 8.18 8.19 9.0", "conclusion": "success", "action_is_running": true}
-*- Mergify Payload End -*-
-->
author_association: CONTRIBUTOR