Repository: kestra-io/kestra
Issue: Allow to search across all Namespace Files (rather than only across currently-open files)
Labels: backend
enhancement
frontend

1. Search by file name
2. Search by flow source

Search by file's content is out of scope for now for several reasons (performance, privacy)

Issue comments:
After discussing with @tchiotludo it was chosen not to include search by flow source for now as it requires some rework around matches retrieval (we need more info on matches' index in flow for VSCode to highlight / show proper preview). Moreover, as we're limited by storages which prevent us from having search by content over Namespace Files (not flows) which in the end leads to incomplete and confusing feature (why my search by content is scoped to flows ?).

I'll only introduce search by file (& flows) name for now.

Pull request review comments:
The second line is misleading, does it retruns path or path relative to the preifx?
I think it should return full URI
So the test shows that it returns relative path which is strange.
Looking at the interface of the LocalStorage I think it should returns a List<URI> instead, containing the full internal storage URI.
This if you need a list of string without the prefix it is something related to NamespaceFiles so it must be adapted inside the NamespaceFile service.
Otherwise, this method will not be reusable for something else.
I don't understand in what this is different from the existing method that list all files for a namespace?
It's recursive. Its goal is to be able to get every file under the given prefix (note the word file, we don't want directories in here) with their path. It walks along the whole hierarchy which is behind the prefix. So for eg. we use it there by giving the NamespaceFiles storage prefix which will allow me to get URIs of every namespace files.

It's required for file search by name in VSCode editor. Ideally we would've liked that the implementation do the search part but it's not feasible as storages don't allow wildcards searching so we would be scoped to "startsWith" alike searches

Following what you said above, I can definitely return the entire URI if that's preferable and do the change in NamespaceFile service
replied in the other comment
> Following what you said above, I can definitely return the entire URI if that's preferable and do the change in NamespaceFile service

Yes, full URIs please, internal storage always use full URIs
You need to name the file differently to check that it is not inside the response from the null tenant
Please add a test for when there is no response (should return an empty list)
Why do you concat with staticFiles and what is this statifFiles?
Nice spot
For easier maintainability and flexibility I added a static file concept for VSCode editor. Those files are served from static resources of the webserver and the goal is to be able as a software to provide some read-only files to the user. This can be used to guide him (for eg. we currently have a getting-started file which appear on VSCode editor but isn't editable, it is served statically and acts like a README for editor usage)
Well, I'm not sure I see the point of including them in the search but anyway, OK
If I don't include them the feature will look unfinished or broken IMO :thinking: 

Patch that resolved the issue:
diff --git a/core/src/main/java/io/kestra/core/storages/StorageInterface.java b/core/src/main/java/io/kestra/core/storages/StorageInterface.java
index 42abe765fd..0640f7b00d 100644
--- a/core/src/main/java/io/kestra/core/storages/StorageInterface.java
+++ b/core/src/main/java/io/kestra/core/storages/StorageInterface.java
@@ -30,6 +30,13 @@ public interface StorageInterface {
     @Retryable(includes = {IOException.class}, excludes = {FileNotFoundException.class})
     InputStream get(String tenantId, URI uri) throws IOException;
 
+    /**
+     * Returns all files that start with the given prefix.
+     * @return Kestra's internal storage uris of the found files
+     */
+    @Retryable(includes = {IOException.class}, excludes = {FileNotFoundException.class})
+    List<URI> filesByPrefix(String tenantId, URI prefix) throws IOException;
+
     @Retryable(includes = {IOException.class}, excludes = {FileNotFoundException.class})
     List<FileAttributes> list(String tenantId, URI uri) throws IOException;
 
diff --git a/storage-local/src/main/java/io/kestra/storage/local/LocalStorage.java b/storage-local/src/main/java/io/kestra/storage/local/LocalStorage.java
index 2f5d9f490c..44a6dcee84 100644
--- a/storage-local/src/main/java/io/kestra/storage/local/LocalStorage.java
+++ b/storage-local/src/main/java/io/kestra/storage/local/LocalStorage.java
@@ -11,6 +11,7 @@
 import java.nio.file.*;
 import java.nio.file.attribute.BasicFileAttributes;
 import java.nio.file.attribute.FileTime;
+import java.util.Collections;
 import java.util.Comparator;
 import java.util.List;
 import java.util.stream.Stream;
@@ -50,6 +51,22 @@ public InputStream get(String tenantId, URI uri) throws IOException {
         );
     }
 
+    @Override
+    public List<URI> filesByPrefix(String tenantId, URI prefix) throws IOException {
+        Path fsPath = getPath(tenantId, prefix);
+        try (Stream<Path> walk = Files.walk(fsPath).filter(Files::isRegularFile)) {
+            return walk.sorted(Comparator.reverseOrder())
+                .map(fsPath::relativize)
+                .map(path -> {
+                    String prefixPath = prefix.getPath();
+                    return URI.create("kestra://" + prefixPath + (prefixPath.endsWith("/") ? "" : "/") + path);
+                })
+                .toList();
+        } catch (NoSuchFileException e) {
+            return Collections.emptyList();
+        }
+    }
+
     @Override
     public boolean exists(String tenantId, URI uri) {
         return Files.exists(getPath(tenantId, uri));
diff --git a/ui/public/init.js b/ui/public/init.js
index 2b91886ff5..770d06bd4e 100644
--- a/ui/public/init.js
+++ b/ui/public/init.js
@@ -89,6 +89,9 @@ window.product = {
                 "saveEditor",
                 "terminalDataWriteEvent",
                 "terminalExecuteCommandEvent"
+            ],
+            "kestra-io.kestra": [
+                "fileSearchProvider"
             ]
         }
     },
diff --git a/webserver/src/main/java/io/kestra/webserver/controllers/NamespaceFileController.java b/webserver/src/main/java/io/kestra/webserver/controllers/NamespaceFileController.java
index 0821a0532b..35acca43f8 100644
--- a/webserver/src/main/java/io/kestra/webserver/controllers/NamespaceFileController.java
+++ b/webserver/src/main/java/io/kestra/webserver/controllers/NamespaceFileController.java
@@ -54,6 +54,21 @@ public class NamespaceFileController {
     );
 
 
+    @ExecuteOn(TaskExecutors.IO)
+    @Get(uri = "{namespace}/files/search", produces = MediaType.TEXT_JSON)
+    @Operation(tags = {"Files"}, summary = "Find files which path contain the given string in their URI")
+    public List<String> search(
+        @Parameter(description = "The namespace id") @PathVariable String namespace,
+        @Parameter(description = "The string the file path should contain") @QueryValue String q
+    ) throws IOException, URISyntaxException {
+        URI baseNamespaceFilesUri = toNamespacedStorageUri(namespace, null);
+        return Stream.concat(
+            storageInterface.filesByPrefix(tenantService.resolveTenant(), baseNamespaceFilesUri).stream()
+                .map(storageUri -> "/" + baseNamespaceFilesUri.relativize(storageUri).getPath()),
+            staticFiles.stream().map(StaticFile::getServedPath)
+        ).filter(path -> path.contains(q)).toList();
+    }
+
     @ExecuteOn(TaskExecutors.IO)
     @Get(uri = "{namespace}/files", produces = MediaType.APPLICATION_OCTET_STREAM)
     @Operation(tags = {"Files"}, summary = "Get namespace file content")
@@ -180,7 +195,7 @@ private void forbiddenPathsGuard(URI path) {
     }
 
     private URI toNamespacedStorageUri(String namespace, @Nullable URI relativePath) {
-        return URI.create(storageInterface.namespaceFilePrefix(namespace) + Optional.ofNullable(relativePath).map(URI::getPath).orElse("/"));
+        return URI.create("kestra://" + storageInterface.namespaceFilePrefix(namespace) + Optional.ofNullable(relativePath).map(URI::getPath).orElse("/"));
     }
 
     private void ensureWritableFile(URI path) {


Test patch:
diff --git a/core/src/test/java/io/kestra/core/storage/StorageTestSuite.java b/core/src/test/java/io/kestra/core/storage/StorageTestSuite.java
index 6aad99adde..5675d94e94 100644
--- a/core/src/test/java/io/kestra/core/storage/StorageTestSuite.java
+++ b/core/src/test/java/io/kestra/core/storage/StorageTestSuite.java
@@ -112,6 +112,41 @@ void getFileNotFound() {
     }
     //endregion
 
+    @Test
+    void search() throws IOException {
+        storageInterface.put(null, URI.create("/namespace/file.txt"), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put("tenant", URI.create("/namespace/tenant_file.txt"), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, URI.create("/namespace/another_file.json"), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, URI.create("/namespace/folder/file.txt"), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, URI.create("/namespace/folder/some.yaml"), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, URI.create("/namespace/folder/sub/script.py"), new ByteArrayInputStream(new byte[0]));
+
+        List<URI> res = storageInterface.filesByPrefix(null, URI.create("kestra:///namespace/"));
+        assertThat(res, containsInAnyOrder(
+            URI.create("kestra:///namespace/file.txt"),
+            URI.create("kestra:///namespace/another_file.json"),
+            URI.create("kestra:///namespace/folder/file.txt"),
+            URI.create("kestra:///namespace/folder/some.yaml"),
+            URI.create("kestra:///namespace/folder/sub/script.py")
+        ));
+
+        res = storageInterface.filesByPrefix("tenant", URI.create("/namespace"));
+        assertThat(res, containsInAnyOrder(URI.create("kestra:///namespace/tenant_file.txt")));
+
+        res = storageInterface.filesByPrefix(null, URI.create("/namespace/folder"));
+        assertThat(res, containsInAnyOrder(
+            URI.create("kestra:///namespace/folder/file.txt"),
+            URI.create("kestra:///namespace/folder/some.yaml"),
+            URI.create("kestra:///namespace/folder/sub/script.py")
+        ));
+
+        res = storageInterface.filesByPrefix(null, URI.create("/namespace/folder/sub"));
+        assertThat(res, containsInAnyOrder(URI.create("kestra:///namespace/folder/sub/script.py")));
+
+        res = storageInterface.filesByPrefix(null, URI.create("/namespace/non-existing"));
+        assertThat(res, empty());
+    }
+
     //region test LIST
     @Test
     void list() throws Exception {
diff --git a/webserver/src/test/java/io/kestra/webserver/controllers/NamespaceFileControllerTest.java b/webserver/src/test/java/io/kestra/webserver/controllers/NamespaceFileControllerTest.java
index 74c334e93a..cbd411488c 100644
--- a/webserver/src/test/java/io/kestra/webserver/controllers/NamespaceFileControllerTest.java
+++ b/webserver/src/test/java/io/kestra/webserver/controllers/NamespaceFileControllerTest.java
@@ -1,5 +1,6 @@
 package io.kestra.webserver.controllers;
 
+import io.kestra.core.serializers.JacksonMapper;
 import io.kestra.core.storages.FileAttributes;
 import io.kestra.core.storages.StorageInterface;
 import io.micronaut.core.annotation.Nullable;
@@ -32,8 +33,7 @@
 import java.util.Optional;
 
 import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.Matchers.containsString;
-import static org.hamcrest.Matchers.startsWith;
+import static org.hamcrest.Matchers.*;
 import static org.hamcrest.core.Is.is;
 
 @MicronautTest
@@ -62,6 +62,28 @@ public void init() throws IOException {
         storageInterface.delete(null, toNamespacedStorageUri(NAMESPACE, null));
     }
 
+    @SuppressWarnings("unchecked")
+    @Test
+    void search() throws IOException {
+        storageInterface.put(null, toNamespacedStorageUri(NAMESPACE, URI.create("/file.txt")), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, toNamespacedStorageUri(NAMESPACE, URI.create("/another_file.json")), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, toNamespacedStorageUri(NAMESPACE, URI.create("/folder/file.txt")), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, toNamespacedStorageUri(NAMESPACE, URI.create("/folder/some.yaml")), new ByteArrayInputStream(new byte[0]));
+        storageInterface.put(null, toNamespacedStorageUri(NAMESPACE, URI.create("/folder/sub/script.py")), new ByteArrayInputStream(new byte[0]));
+
+        String res = client.toBlocking().retrieve(HttpRequest.GET("/api/v1/namespaces/" + NAMESPACE + "/files/search?q=file"));
+        assertThat((Iterable<String>) JacksonMapper.toObject(res), containsInAnyOrder("/file.txt", "/another_file.json", "/folder/file.txt"));
+
+        res = client.toBlocking().retrieve(HttpRequest.GET("/api/v1/namespaces/" + NAMESPACE + "/files/search?q=file.txt"));
+        assertThat((Iterable<String>) JacksonMapper.toObject(res), containsInAnyOrder("/file.txt", "/folder/file.txt"));
+
+        res = client.toBlocking().retrieve(HttpRequest.GET("/api/v1/namespaces/" + NAMESPACE + "/files/search?q=folder"));
+        assertThat((Iterable<String>) JacksonMapper.toObject(res), containsInAnyOrder("/folder/file.txt", "/folder/some.yaml", "/folder/sub/script.py"));
+
+        res = client.toBlocking().retrieve(HttpRequest.GET("/api/v1/namespaces/" + NAMESPACE + "/files/search?q=.py"));
+        assertThat((Iterable<String>) JacksonMapper.toObject(res), containsInAnyOrder("/folder/sub/script.py"));
+    }
+
     @Test
     void file() throws IOException {
         String hw = "Hello World";