From Commands to Prompts: LLM-based Semantic File System

## Abstract

Abstract Large language models (LLMs) have demonstrated significant potential in the development of intelligent applications and systems such as LLM-based agents and agent operating systems (AIOS). However, when these applications and systems interact with the underlying file system, the file system still remains the traditional paradigm: reliant on manual navigation through precise commands. This paradigm poses a bottleneck to the usability of these systems as users are required to navigate complex folder hierarchies and remember cryptic file names.
To address this limitation, we propose an LLM-based semantic file system ( LSFS ) for prompt-driven file management. Unlike conventional approaches, LSFS incorporates LLMs to enable users or agents to interact with files through natural language prompts, facilitating semantic file management. At the macro-level, we develop a comprehensive API set to achieve semantic file management functionalities, such as semantic file retrieval, file update monitoring and summarization, and semantic file rollback). At the micro-level, we store files by constructing semantic indexes for them, design and implement syscalls of different semantic operations (e.g., CRUD, group by, join) powered by vector database. Our experiments show that LSFS offers significant improvements over traditional file systems in terms of user convenience, the diversity of supported functions, and the accuracy and efficiency of file operations. Additionally, with the integration of LLM, our system enables more intelligent file management tasks, such as content summarization and version comparison, further enhancing its capabilities.
The code is open-source at https://github.com/agiresearch/AIOS-LSFS .

## 1 Introduction

In recent years, researchers have integrated AI with operating systems to create innovative, efficiency-boosting applications ; . For example, machine learning algorithms can dynamically adjust system resource allocation ; . Besides, by monitoring application performance and user behavior, AI models can predict future resource needs and pre-allocate CPU, memory, and other resources, enhancing overall system performance and response speed . With the recent developments of generative AI, researchers have envisioned Large Language Models (LLMs) based operating systems .

At present, LLMs have demonstrated great planning and reasoning ability ; ; ; ; ,
which facilitates the development of LLM-based agents, including single-agent applications ; ; ; and collaborative multi-agent applications ; ; ; . Nowadays, researchers have been trying to embed LLMs from the application-level to the system-level to serve users and AI agent applications, enabling LLM-based artificial intelligent operating system (AIOS) . Through such integration, the powerful understanding and generation ability of LLMs not only can help to complete tasks on the application-level, but also can help to intelligently manage various resources such as memory and storage on the system-level.

File system is one of the most fundamental and essential components of an operating system, yet it comes with significant limitations in terms of AIOS. Currently, file management primarily relies on index structures that organize files based on attributes like file name, size, and timestamps. However, this approach does not leverage the semantic meaning of the files themselves. ❶ For instance, if two files have similar content–such as different versions of the same document–traditional file systems lack the capability to organize or retrieve these files based on their content similarity.
❷ User interactions with traditional file systems require complex operating system commands or manual navigation through the user interface, forcing users to precisely recall file names or locations. For systems with numerous files, this retrieval process can be inefficient and time-consuming, reducing overall system usability. Nowadays, based on the strong language understanding capability of LLMs, we can make better use of the file content and semantic information for file management by introducing LLMs into the system. However, existing works on using LLMs to facilitate file management are mostly conducted on the application level, which targets at designing specific agent for file retrieval and manipulation ; . The community still lacks a more general LSFS to serve as a common foundation that can be used by various agents on the application-level.
To solve the above problems, we propose LSFS, an LLM-based semantic file system, to enhance the interaction between LLM and traditional file system for better semantic file management.

For problem ❶, our LSFS introduces a semantic-based index structure that leverages a vector database for file storage. By extracting semantic features from the file content and generating corresponding embedding vectors, LSFS incorporates semantic information into its file operations. Additionally, we have designed numerous reusable syscall interfaces for LSFS, modeled after traditional file system functions. At the same time, we design several APIs that can realize complex file functions based on the syscalls. These syscalls and APIs not only can realize the basic functions of the file system but also can provide the operations that the traditional file systems do not include.

Figure: Figure 1: The pipeline of changing file function in traditional system and our file management system
Refer to caption: x1.png

To address problem ❷, we integrate LLM into the API for complex functions and introduce a template system prompt. This allows us to utilize LLM to extract keywords from the user’s natural language input and map them directly as input parameters for the API or syscall, streamlining the interaction between users and the system. The comparison of commands executed by users in traditional file systems and LSFS is shown in  Figure 1. In  Figure 1(a), When a user wishes to modify a file’s content, they must input a specific command in the terminal, requiring them to remember the correct operator and the exact paths for both the target and source files. This approach places a significant burden on the user’s memory.
However, LSFS can effectively solve this problem. Users only need to manage files by typing a natural language prompt as a command. For example, as shown in Figure Figure 1(b), the user needs to input a simple natural language description, our LSFS is able to understand prompts and perform the corresponding operation. This approach greatly simplifies the complexity of the user’s operation and improves the efficiency of the user’s operation. Furthermore, to reduce the LLM’s hallucination problem of generating inaccurate instructions, especially those irreversible operations, we design systematic safety insurance mechanisms in LSFS, such as safety checks for irreversible operations and user verification before instruction execution.

Overall, our research contributes as follows:

- •
We introduce a LLM-based Semantic File System (LSFS) to manage files in a semantic way. By altering the file storage structure and method, LSFS incorporates the semantic context of files, optimizing the fundamental functions of traditional file systems. Additionally, we develop a variety of reusable syscalls and APIs within LSFS, allowing for extended functionality and enabling future developments based on this system.
- •
Our system provides a LSFS parser, which can parse natural language prompts into executable APIs supported by LSFS, enabling the execution of relevant file management tasks. This allows users to control and manage files using simple natural language prompts, acting as a bridge that translates user/agent instructions into system actions. By doing so, this approach greatly simplifies file system operations and enhances overall efficiency.
- •
To avoid unintended operations in LSFS, especially those irreversible operations, we design systematic safety insurance mechanisms, such as safety checks for irreversible operations and user verification before instruction execution, ensuring the safety and accuracy of LSFS.
- •
Through our experiments, we validate the completeness of LSFS’s functions, while also demonstrating its innovation and superiority in handling specific tasks compared to traditional file systems. Furthermore, the experiments suggest that the LSFS enhances the execution of file management operations, making it more intelligent and efficient.

## 2 Related Work

### 2.1 Semantic File System

Currently, file storage and retrieval primarily rely on an index structure maintained by the system, where file metadata points to the file’s location on the disk . While optimizing the index structure can enhance retrieval efficiency, the current storage model is still largely dependent on the keywords extracted from the file’s content. Gifford et al. were the first to propose a semantic file system, which introduced a layer that generates directories by extracting attributes from files, enabling users to query file attributes through navigation. proposed a semantic file system to manage the data. Many subsequent works have integrated semantics into metadata ; ; ; ; . Mahalingam et al. proposed a semantic deep file management system called Sedar, which embeds semantic features into file metadata, allowing retrieval to be matched with semantic vectors. However, Sedar faces challenges with the similarity of extracted semantic vectors. Hua et al. further integrated semantic information into metadata, restricting complex queries to a related metadata group. The same authors also utilized semantics to reduce the relevance of queries using the semantic similarity between files based on the semantic naming system. Leung et al. used semantic information combined with the file system design of graphs to provide scalable search and navigation. On the other hand, Bloehdorn et al. proposed to manage files through semantic tags. Schandl et al. developed an approach for managing desktop data using a semantic vocabulary. In contrast, our semantic file system is based on the strong language understanding ability of LLMs. Besides, it integrates comprehensive semantic information across all aspects of the file system–from storage and file operations to practical applications. This holistic approach enhances the system’s ability to understand and manage files, significantly improving functionality beyond what traditional systems and earlier semantic file systems offer.

### 2.2 Semantic Parser

Researchers have also devoted efforts to developing semantic parsers ; ; ; ; capable of transforming natural language into a machine-interpretable format. Mooney et al. pioneered this area by introducing a suite of machine learning-based semantic parsers designed for specific tasks. Iyer et al. subsequently focused on parsing database commands, while Berant et al. proposed a question-answer pair learning approach to enhance parsing efficiency. In further work, the same authors explored a paraphrasing technique to improve semantic parsing performance.
Poon et al. introduced a Markov logic-based approach, and Wang et al. addressed the challenge of building parsers from scratch in new domains. Ge et al. proposed a parse tree-based method for more accurate semantic analysis. Notably, Lin et al. were the first to integrate a semantic parser into an operating system, leveraging a dataset of bash commands and expert-written natural language to establish a mapping between the two. However, this approach faced limitations in handling complex semantics and unseen natural language.
In contrast, our LSFS is built upon more constructing more semantic index as the format of embedding vectors, improving its generation ability to understand and process diverse natural language inputs.

### 2.3 OS-related LLMs and Agents

To enhance user experience and improve task completion efficiency, many LLM-based AI agents have been introduced to fulfill operating system related tasks such as software development ; and manipulating resources on traditional OS ; .
Wu et al. developed LLM-based agents for co-piloting users’ interaction with computers, such as drawing charts and creating web pages. MetaGPT employs a sophisticated large language model in a multi-agent conversational setting to automate software development, assigning specific roles to various GPTs for seamless collaboration.
CHATDEV presents a framework for software development, utilizing agents to enhance cooperation among different roles within the development process. Beyond the application-level research on LLM-based agents, researchers also explored integrating LLMs into the system-level ; , which enables LLM-based agent operating system (AIOS) that provides fundamental services to the agent applications on top of it, such as agent scheduling, memory and storage management, tool management, security features, and access control.

## 3 Architecture of LSFS

Traditional file systems primarily rely on file attributes to build metadata. These attributes, typically obtained by scanning the file, include file size, creation and modification timestamps. The actual file content is stored as binary data, with traditional file systems leveraging index structures such as B+ trees to efficiently locate this data. While these designs continue to evolve and improve, they generally overlook the semantic content information within files, making it difficult for traditional file systems to support tasks that require deeper semantic understanding. To address this limitation, we propose a semantic file management system powered by LLMs and vector databases.

Figure: Figure 2: (a) provides a overview of the LSFS architecture, and (b) shows the internal APIs and syscalls in LSFS.
Refer to caption: x2.png

This system retains traditional file attributes and integrates semantic information derived from the file’s content. Unlike traditional file systems, our system enhances metadata by including user-defined keywords that capture the essence of the file’s content, in addition to standard file attributes. Our semantic file management system employs embedding models to convert textual content into semantic vector representations. These vectors enable the construction of a semantic-based data index, facilitating the structured storage of data and enabling robust, semantically-driven file retrieval and management. This optimized architecture allows for more intelligent and semantic interactions with file data, enhancing the system’s functionality and making it particularly well-suited for modern file management tasks.

Figure 2(a) outlines the overall architecture of our LSFS. LSFS operates as an additional layer on top of traditional file systems, which constructs semantic index of the file in the format of embedding vectors, thereby enabling more advanced content-based file management operations. LSFS provides a range of system calls (syscalls) that support complex file operations and reuse. The system also includes a supervisor that works in tandem with the traditional file management system. This supervisor continuously monitors changes in the traditional file system and synchronizes those changes in LSFS, ensuring that the contents of LSFS remain consistent with the underlying file system.  Figure 2(b) presents an overview of the syscall structure in LSFS. These syscalls are primarily categorized into two types: atomic syscalls and composite syscalls. The former handles basic operations such as adding, deleting, modifying, and searching files, while the latter builds upon atomic syscalls to execute more complex functions. On the top of the syscalls, LSFS APIs are constructed by utilizing one or more syscalls and integrating LLMs. These APIs facilitate more advanced yet general-purpose file operations, demonstrating the superiority of LSFS in handling such tasks. Further details on these capabilities will be discussed in Section 4.3.

## 4 Implementation of LSFS

In this section, we introduce our implementation of the LSFS. We present the key functions implemented in our LSFS and compare the counterparts with traditional file systems, which can be seen from the Table 1.

Table: Table 1: Comparison of some key functions between our LSFS and traditional file system (TFS).

We introduce the implementation of LSFS from the bottom to the top in the LSFS architecture shown in the Figure 2. We start by introducing the basic syscalls implemented in LSFS and introduce the supervisor which interacts between LSFS syscalls and traditional file systems. Then we present the APIs that built upon the syscalls to achieve more complex functionalities. After that, we introduce the LSFS parser on top to show how natural language prompts have been decoded into executable LSFS APIs. At last, we use different concrete prompts to show how different modules in the LSFS are executed to achieve functionalities.

### 4.1 Basic Syscall of LSFS

In this section, we introduce the syscalls implemented for LSFS. These syscalls are primarily categorized into two types: atomic syscalls and composite syscalls. Atomic syscalls involve operations covering the most basic operations, e.g., create, retrieve and write of files. Composite syscalls are combinations of two or more atomic syscalls to execute composite functions, e.g., join and group by.

#### Atomic Syscall of LSFS .

These syscalls involve the atomic operations that cannot be divided further into sub operations, i.e., creation, retrieval, write, and deletion of files.

- •
create_or_get_file() This syscall integrates various functions of traditional file systems, including creating, reading, and opening files, and performs specific operations based on the provided parameters. The return value of this syscall can be used to retrieve file metadata, modification timestamps, the file’s memory path, and other essential information.
- •
add_() This syscall is used to write new content to the end of a specified file within the LSFS.
- •
overwrite() This syscall is used to overwrite the contents of the original file with the new file and generate new metadata for this file as required by the user.
- •
del_() This syscall is designed to delete specified files and offers two methods of deletion. First, it allows deletion by specifying the file name or file path. Second, it supports keyword-based deletion, identifying and removing files that contain a given keyword. Additionally, if all files within a directory are deleted, the syscall automatically remove the directory itself.
- •
keywords_retrieve() This syscall is used to implement a keyword search function that retrieves files containing a keyword in a specified directory. It supports single condition matching and multi-condition matching, and returns the filename and file contents.
- •
semantic_retrieve() This syscall is used to implement the semantic matching function to retrieve the top-n highest semantic similarity files in directory and retrieval conditions according to the similarity score. It returns the filename and file contents.

#### Composite Syscall of LSFS .

These syscalls involve composite operations that are built by combining two or more atomic syscalls to perform operations.

- •
create() This syscall is used to create files in bulk in LSFS by importing the path of the folder in memory and importing all the files in the folder under the corresponding directory.
- •
lock_file() / unlock_file() The two syscalls are used to lock/unlock a file by changing the file state to read-only via lock_file and changing the file permission to read-write via unlock_file.
- •
group_semantic() The syscall can select the content in the specified directory and retrieve the files that have high similarity with the query, create a new directory, and place the selected files in the directory to facilitate the operation of the files that have the same subject.
- •
group_keywords() This syscall can select the files that contain the retrieved keywords in the specified directory, create a new directory, and place the selected files in the directory to facilitate the operation of the files that contain the same keywords.
- •
integrated_retrieve() This syscall combines two retrieval methods to retrieve the files that contain a particular keyword and that are similar in content to the retrieval query. The order of retrieval is keyword search first, and then semantic search.
- •
file_join() This syscall can be used to concatenate two files into a single file, either by creating a new file to concatenate or by concatenating the original file directly.

### 4.2 Supervisor

The supervisor is implemented to track the changes in the files in the disk and sync the changes to the LSFS.
The supervisor periodically scans the files within its specified directory. When it detects any change or deletion of the file content, it automatically synchronizes this information with the LSFS by invoking the appropriate syscall. This ensures that the state of the file in the LSFS reflects the current state of the file in memory. In order to ensure the accuracy of other syscall operations on the file after Supervisor modifies the internal file status of LSFS, LSFS introduces the process lock mechanism to ensure that multiple processes can access the file correctly. When asked, the supervisor can generate status change reports. For example, when a file is modified, the supervisor invokes LLMs to generate a detailed modification log, compares the contents of the file before and after modification, and presents this log to the user.

### 4.3 API of LSFS

In this module, we introduce the APIs that are implemented on top of the syscalls mentioned in Section  2 to support higher-level semantic file management functions. Specifically, we provide the following APIs that cover the basic semantic file management requirements, i.e., semantic CRUD (create, read, update, and delete) of files.

#### Retrieve-Summary API.

This API retrieves files based on user-specified conditions and provides concise summaries. Unlike traditional systems, it offers both keyword and semantic search, along with LLM-powered content summarization. The API supports three retrieval methods: keyword, semantic, and integrated, which are built on top of the keywords_retrieve(), semantic_retrieve(). and integrated_retrieve() syscalls.
In this API, an interaction interface is also provided for users to refine the results by excluding irrelevant files, which are then passed to LLMs for summarization.

#### Change-Summary API.

This API is used to modify the contents of a file and compare them before and after to summarize the changes.
At the same time, the Supervisor module is introduced to monitor the file changes in the traditional file system. Unlike traditional file systems that require tedious operating system commands, this API allows users to locate target files using natural language and automatically generate a summary of file changes through LLMs integration.
The API is implemented by leveraging the supervisor and the overwrite() and del_() syscalls.
In the change-summary API, when the target file is updated with the content of the source file, it will generate a summary of the modifications. Meanwhile, the filename is used as the key in the version recorder, while the metadata and contents of the file are stored as the corresponding values for the version control use.

#### Rollback API.

This API is designed to achieve version restoration by utilizing the version recorder from the change-summary API, making version rollbacks more manageable. In this API, the overwrite() and create_or_get_file() syscalls are employed. Traditional file systems like ZFS, BtrFS, and NTFS offer rollback capabilities, but they primarily rely on snapshots, where the system captures the state of files at a specific point in time and restores them to that version. Our rollback API in the LSFS introduces two rollback methods for greater flexibility and ease of use. The first is time-based rollback, where the LLMs parses the rollback time from the user’s input and reverts the file to the corresponding version. The second is version-based rollback, allowing users to specify in the prompt how many versions backward they wish to revert. This dual approach makes it easier for users to rollback to the target version of files.

#### Link API.

This API is designed to enable the creation of shareable file links. In traditional file systems, links can only be generated for local access, limiting collaboration. However, with the LSFS’s link API, shareable links can be generated for broader accessibility.
Specifically, the API leverages cloud database, e.g., Google Drive, to upload files and generate the shareable link.
Additionally, validity period can be passed as an argument for this API, once the period expires, the link API automatically revokes the link and terminates access for secure and time-bound file sharing.

### 4.4 LSFS Parser

Figure: Figure 3: The example of using LLMs to extract the key information from natural language prompt.
Refer to caption: x3.png

We design an LSFS parser to guide this conversion process and create a general prompt template for future API designs to extract parameters from natural language prompts. As illustrated in the accompanying  Figure 3, we leverage large language model to parse natural language commands. For each API, we design distinct system prompt words, which, when input alongside the user’s command, allow the language model to automatically detect and extract the key parameters, outputting them in a comma-separated format. By applying a simple regex to this output, we can directly convert the extracted keywords into syscall parameters, enabling seamless execution of the API command. This allows us to efficiently map natural language commands to the exacted parameters required by the system calls, benefiting interactions between natural language prompts and LSFS.

Figure: Figure 4: Details of different API callings inside the LSFS. In this figure, (a)-(d) show interactive examples of how LSFS solves file management tasks step by step.
Refer to caption: x4.png

### 4.5 The Interaction between Modules

In Figure 4, we present the examples to demonstrate how components of LSFS interact with each other to achieve different functionalities. The upper section of Figure 4 depicts the workflow of the retrieve-summary API, while the lower section outlines the workflows of the change-summary API and rollback API.
In the upper part of Figure 4, the LSFS parser decodes prompts into API calls with API name and API arguments. Then LSFS executes the API to check vector database to get results. This API also provides user-interaction interface for the users to verify results. After the verification, the content will be summarized by leveraging LLM.
In the lower part of Figure 4, when a modification request is submitted, the LSFS parser decodes the file information (name and location) that is to be modified. The LSFS then modifies the semantic changes in both the vector database and the files stored in the disk. Meanwhile, the supervisor of the LSFS is kept running to ensure consistency between the semantic index of files in the LSFS and the files stored in the disk. Upon updated, the summarization API compares the file contents before and after the change to generate a detailed change log. Additionally, the API stores the pre-modification content in the version recorder. If a rollback is requested, the API retrieves the specific version from the version recorder and synchronizes it in both the LSFS and files in the disk.

## 5 Evaluation

In this section, we propose the following research questions regarding the performance of LSFS and conduct experiments to answer these research questions.

- •
RQ1: What is the success rate of the LSFS parser to parse natural language prompts into executable API calls?
- •
RQ2: How does LSFS perform in semantic file management tasks, e.g., semantic file retrieval and semantic file rollback?
- •
RQ3: Can LSFS still maintain good performance in non-semantic file management tasks, such as keyword-based file retrieval and file sharing?

### 5.1 RQ1: Effectiveness of LSFS Parser

For RQ1, we assess the accuracy of the LSFS parser in translating user natural language prompt into executable LSFS API calls. We evaluate the accuracy of LSFS parser with 30 different samples for each API on different LLM backbones, i.e., Gemmi-1.5-Flash, GPT-4o-mini, Qwen-2, and Gemma-2. The results, illustrated in Figure 5, reveal that the LSFS parser performs exceptionally well on parsing prompts related to change-summary API and link API (for which the semantic information in the user prompt is relatively simple), achieving high accuracy across all LLMs, with some reaching 100%

Figure: Figure 5: The accuracy of LSFS parser in translating natural language prompt to executable API calls.
Refer to caption: x5.png

accuracy. For more complex prompts, such as those intended for the rollback API and retrieve-summary API (for which the semantic information in the user prompt is complex), accuracy remains above 85% for most models, except for Gemma-2.
The average parsing accuracy reaches 90%. These results show that the LSFS parser can effectively translate natural language information into executable API calls, showcasing its reliability in diverse scenarios. In all cases, the translated command is provided to users for approval before real execution, ensuring safety when executing commands in the file system.

### 5.2 RQ2: Analysis of LSFS in Semantic File Management Tasks

#### Performance Analysis in Semantic File Retrieval .

Table: Table 2: Comparison of the accuracy and execution time between using LSFS and the baseline which incorporates LLM into traditional file system without using LSFS.

In our experiments, we compare the performance of using LSFS and without using LSFS under the same LLM backbone.
The details of the prompts we use for the comparison are in the Appendix  B.
Specifically, we use Gemini-1.5-flash and GPT-4o-mini as the LLM backbone, respectively, for the comparison. As shown in Table 2, using LSFS to implement the retrieval function significantly enhances both the accuracy and the efficiency compared to only leveraging LLM for retrieval without using LSFS. As file number increases, the retrieval accuracy tends to drop significantly when using LLM for retrieval without LSFS. This is because more files can lead to longer context for the LLM, which degrades the LLM’s performance of identifying information in the long context.
By contrast, using LSFS can still achieve good retrieval accuracy and have much better retrieval efficiency when file number increases, because lsfs replaces the reasoning process of LLM by using keyword matching and semantic similarity matching, it saves a lot of time and avoids the errors of LLM when facing complex input text.

Figure: Figure 6: A file summarization agent instance to illustrate the comparison between LLM-leveraged methods without using LSFS and using LSFS.
Refer to caption: x6.png

We conduct a case study using the example prompt “Please search for the two papers most related to LLMs Uncertainty from folder named example” to better illustrate the retrieval results, which is shown in the Figure 6.
For the method without using LSFS, the answer to the intermediate result is “GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning$\cdots$,” which fails to identify a target paper that is relevant to LLMs Uncertainty due to the long-context that makes it difficult for LLM to understand and retrieve the correct information. Furthermore, the input length limitation of LLM necessitates a batch-processing strategy for file input, which can result in selecting the “best of the worst” candidates. This may lead to inaccurate intermediate results that ultimately impact the final output. In contrast, LSFS avoids this issue by evaluating all files holistically, scoring and sorting the final results without intermediate outputs, thus bypassing the constraints of limited search scope. Experimental results show that LSFS consistently delivers more accurate results.

#### Scalability Analysis in Semantic File Rollback .

Figure: Figure 7: The relationship between the number of versions of a rolled back file and the rollback time.
Refer to caption: x7.png

LSFS supports semantic file rollback, which enables the restoration of a file to a particular version specified by the time requested by the user or number of versions, recorded by the Version Recorder. We vary the the number of rollback versions and calculate their corresponding rollback time, to evaluate the stability and efficiency of the version rollback process,
The results, shown in Figure 7, illustrate the consistency in the time consumed during version rollbacks.
We use Gemmi-1.5-Flash, GPT-4o-mini, Qwen2 as the LLM backbones for the experiments. In our experiment, we set the file version rollback process to begin from the newest one version, with a rollback occurring every last 5 versions. The experiment continues until a maximum of 40 versions had been rolled back. Each rollback is simultaneously updated in both the LSFS and file of the computer’s local disk. As shown in the Figure 7, across all three LLM backbones, the rollback time does not increase exponentially with the number of versions rolled back. Instead, it tends to plateau. This is because the rollback API stores each version of the file independently, thereby avoiding long rollback paths to implement rollbacks and achieving a stable rollback time.

### 5.3 RQ3: Analysis of LSFS in Non-Semantic File Management Tasks

For RQ3, we evaluate on non-semantic file management tasks to measure whether LSFS can still maintain good performance as traditional file systems in these tasks.

#### Performance Analysis in Keyword-based File Retrieval .

In this section, we compare LSFS and traditional file system in keyword-based file retrieval task. The task is to use keywords existing in the filename or file content to retrieve files. We build a hierarchical file folder with file numbers as 10, 20, and 40, respectively, for this task. We use two types of retrieval prompts, i.e., single-condition and multi-condition, to evaluate LSFS and traditional file systems to retrieve relevant files containing specific keywords. The details of how we construct prompts for this task can be seen at Appendix B. We consider the following methods as the retrieval baselines in the traditional file system.
It is important to note that the original grep command can only deal with plain text files, such as .txt and .md files, and cannot support binary files, such as .pdf and .doc files. Therefore, we make two enhanced versions, named as TFS-grep and TFS-grep* to make the comparison.

- •
TFS search window: We use the default search window in the computer’s file folder to retrieve files (i.e., Spotlight in MacOS) which supports retrieving by keywords in the file content.
- •
TFS-grep: We use Python program to convert the binary file to a plain text file and then perform grep operation on the converted plain text file.
- •
TFS-grep^*: After converting binary files into plain text, issues such as missing spaces, incorrect line breaks, and formatting errors may arise. In the TFS-grep^* process, we correct the format of the converted files and then run the grep operation on the properly formatted versions.

**Table 3: Comparison between LSFS and methods in the traditional file system (TFS) in retrieving files by keywords in the names and content of files.**
| Metric | # files | TFS search window | TFS-grep | TFS-grep^* | LSFS |
| --- | --- | --- | --- | --- | --- |
| Precision | 10 | 0.708 | 0.389 | 1.000 | 0.950 |
| 20 | 0.724 | 0.396 | 1.000 | 0.870 |  |
| 40 | 0.691 | 0.403 | 1.000 | 0.863 |  |
| Recall | 10 | 1.000 | 0.416 | 1.000 | 0.833 |
| 20 | 1.000 | 0.292 | 1.000 | 0.933 |  |
| 40 | 1.000 | 0.306 | 1.000 | 0.960 |  |
| F1-score | 10 | 0.829 | 0.402 | 1.000 | 0.891 |
| 20 | 0.840 | 0.337 | 1.000 | 0.900 |  |
| 40 | 0.817 | 0.348 | 1.000 | 0.909 |  |

We use precision, recall and F1-score to measure the retrieval performance.
From the results presented in Table 3, we can find that LSFS outperforms TFS search window and TFS-grep, only second to the TFS-grep*.
We find that the built-in retrieval tool in the TFS (e.g., the system search window) can not generate stable retrieval results, although it has a higher recall.
Due to the fuzzy search feature in the built-in search window, it can easily retrieve inaccurate results for which the retrieved file content can only match part of the keywords. For example, if we search for an article written by John Smith, any other article with a name of John can be returned as a search result, thus the results may often include many irrelevant results, which complicates the process for users to filter through them.
The TFS-grep* command, although achieving perfect results, still has several limitations. First, the commands can be too difficult to construct, especially when the file retrieval queries have multiple conditions. For instance, when a user requests to retrieve all files containing the keywords A and B, the command would be as follows: find /path -type f -exec grep -l "keyword1" $\backslash$; -exec grep -l "keyword2" $\backslash$;. Second, since the grep command itself does not support retrieval of binary files,
it is necessary to manually adjust the format of each file, which is time-consuming and greatly reduces the efficiency of the retrieval process for users. Our LSFS can retrieve all types of text files, from plain text to binary files, while maintaining high precision and recall. The LSFS read operation is capable of processing both plain text and binary text, converting them into the system’s vector database. This enables seamless retrieval operations across various types of files.
Furthermore, LSFS allows users to describe their retrieval tasks in natural language, eliminating the need to write complex commands.

#### Performance Analysis in File Sharing .

For the file sharing task, we evaluate whether a system can output a shareable link with an expiration time according to the prompts.
Specifically, we compare LSFS with four different baselines as following.

- •
Gemini-1.5-flash: We use Gemini-1.5-flash as the LLM backbone to write the code that generates the link for the target file, and then use the Python compiler to check the validity of the code.
- •
GPT-4o-mini: We employ GPT-4o-mini as the LLM backbone to generate code for creating links of the target file, followed by using the Python compiler to verify the code’s validity.
- •
AutoGPT: We create a CoderGPT agent by initializing GPT as an expert on coding, which is used to generate the code.
- •
Code Interpreter: We use the Code Interpreter module of the OpenAI web client to generate relevant code and subsequently check the validity of the code.

Table: Table 4: Comparison between LSFS and other LLM-leveraged methods in File Sharing.

In the experiments, four key metrics are used to evaluate the effectiveness of whether the system can successfully fulfill the file sharing task: whether the LLM generates code, the correctness of the generated code, the effectiveness of the generated links, and whether the links are actually shareable. We evaluate with 20 file sharing task prompts for all the methods. The results show that although all methods even vanilla LLMs can successfully generate code, they do not consistently generate valid links. In many cases, these links are local rather than shareable links, and only a small fraction of the links for files are shareable. In contrast, our LSFS system achieves 100% link generation success rate, showing strong task fulfillment ability on the file sharing task.

## 6 Conclusions

In this paper, we present an LLM-based semantic file system (LSFS), which offers advancement over traditional file systems by enabling files to be stored and managed based on their semantic information. This enhancement improves the system’s ability to comprehend and utilize the semantics embedded in file contents. Additionally, we introduce a series of reusable semantic syscalls and a framework for mapping natural language into LSFS parameters. These innovations provide a foundation for future research and development in the area of semantic file management systems. Looking ahead, future work can further explore and expand the application of LSFS across a wider scope of user environments and integrate semantic file management into everyday computing. This can help to simplify user operations and pave the way for a more intelligent and user-friendly operating system.

## 7 Acknowledgement

We thank Minghua Ma and Sam Lin for the useful discussions with them.

## Appendix

## Appendix A The Implementation Details of Syscall

create_or_get_file() This syscall integrates multiple traditional file system functions such as creating files, reading files, and opening files, enabling different operations based on the parameters provided. The function accepts four parameters: the LSFS path, the target directory name, the target filename, and the import file. The first two parameters are positional, while the latter two are default parameters. When all four parameters are provided, if the target file does not exist within LSFS, the system will create an imported file using the specified target directory name, target filename, and the import file. The import file can be supplied as a string or a file path, and our system supports various text file formats, including PDF, DOCX, TXT and so on. If the target filename is not passed, the syscall returns a list of files within the target directory. If the content of the import file is not provided, the syscall will return the target file, allowing access to its content, metadata, embedding vector, and other associated information.

add_() This syscall facilitates appending content to a file by accepting four positional parameters: the LSFS path, the target directory name, the target filename, and the content of the import file. The import file content can be provided either as a string or a text file in various formats. When all four parameters are supplied, the syscall appends the specified content to the designated file within the system.

overwrite() This syscall implements the overwriting of the file contents. The passed Parameters are also LSFS path, target directory name, target filename, import file, all of which are Positional Parameters. When passed in, LSFS will overwrite everything in the source file with the contents of the imported file.

del_() This syscall performs the deletion of files and directories by accepting four parameters: the LSFS path, the target directory name, the target filename, and the key text. The first two parameters are positional, while the last two are default parameters. If neither of the last two parameters is provided, the syscall raises an error, indicating that at least one must be passed. When the target filename is provided, LSFS deletes the specified file. If the key text is provided instead, the system searches for files containing the key text within the target directory and deletes them if found. Additionally, once all files within a directory are deleted, LSFS will automatically remove the directory.

keywords_retrieve() This syscall implements LSFS’s keyword search functionality, retrieving all files within a specified directory that contain a given keyword. The arguments passed include the LSFS path, directory name, keyword, and matching condition. The LSFS path and keyword are Positional Parameters, while the directory name and condition are Default Parameters. If a directory name is provided, the syscall retrieves files within that directory that match the keyword; otherwise, it searches across the entire system. To match multiple keywords, the matching condition must be passed, specifying the relationship between keywords, such as and or or. The search results return a list of file names and a list of file contents.

semantic_retrieve() This syscall implements the semantic similarity search function within the LSFS, allowing retrieval of files that semantically match a given query within a specified directory. The parameters for this function include the LSFS path, the target directory, the search keyword, and the number of results to return. The LSFS path and search keyword are Positional Parameters, while the target directory and number of results are Default Parameters. Similar to keyword-based retrieval, the search directory will be determined based on whether a target directory is provided. The number of results to return dictates how many of the top-scoring matches are retrieved. Our semantic retrieval leverages the LlamaIndex framework. During file creation, a LlamaIndex vector store is generated alongside the index, enabling more intelligent and efficient data retrieval. This setup ensures that semantic queries can return highly relevant results with improved accuracy. The search results return a list of file names and a list of file contents.

create() This syscall function facilitates batch directory creation and bulk file reading. It accepts the LSFS path, directory name, and the import file path as Positional Parameters. This function can read multiple files at once and store them in the specified target directory. If the filenames are not explicitly provided, they will default to the original filenames from the filesystem.

lock_file() / unlock_file() These syscalls handle the locking and unlocking of files within the LSFS, allowing files to be placed in read-only mode to prevent modification. Both syscalls accept the LSFS pathname, directory name, and filename as parameters. Upon execution, the lock_file() syscall updates the file’s metadata to reflect a read-only state, effectively restricting any modifications. Conversely, the unlock_file() syscall modifies the metadata to restore read and write permissions, enabling the file to be edited again. These operations provide granular control over file access and modification rights.

group_keywords() This syscall groups all files containing a common keyword and creates a new directory for them. The parameters passed are the LSFS name, the keyword, the name of the new directory, the target directory, and the search criteria. Among these, the LSFS name, keyword, and new directory name are Positional Parameters, while the target directory and search criteria are Default Parameters. The syscall first performs a search using the keyword through an atomic syscall to identify all matching files. It then uses these files to create the specified new directory, facilitating easier file management and organization.

group_semantic() This syscall facilitates the organization of files by grouping all those containing a common keyword into a new directory. It takes the following parameters: the LSFS path, the keyword, the new directory name, the target directory, and the search criteria. Here, the LSFS path, keyword, and new directory name are Positional Parameters, while the target directory and search criteria are Default Parameters. The syscall first performs a keyword search through an atomic syscall to identify all files that match the keyword. It then creates a new directory with the specified name and moves the identified files into this directory. This functionality streamlines file management and enhances organizational efficiency.

integrated_retrieve() This syscall is designed for composite searches, combining both semantic and keyword search functionalities. The parameters are distributed as follows: the LSFS path, keyword, new directory name, and search criteria are Positional Parameters, while the target directory and additional search conditions are Default Parameters. The syscall first invokes the keyword grouping function to retrieve all files associated with the compound keyword and organizes them into a new directory. It then performs a semantic search within this directory, ultimately returning the filenames and contents of the files that match the search criteria. This integrated approach allows for more comprehensive and flexible search capabilities.

file_join() This syscall facilitates the connection of two files, with the parameters including the LSFS path, the directory name and filename of both files, and the connection conditions. The Default Parameters are the destination directory for file 2 and the connection condition, while the LSFS path, directory names, and filenames are Positional Parameters. If the destination directory for file 2 does not exist, the two files will be placed in the same directory. If the destination directory exists, the files will be placed in their respective directories. If the join condition is set to new, the syscall will preserve the original files, concatenate their text contents, and create a new file in the destination directory of file 1. The new file’s name will be a combination of the two target filenames. If the join condition is not new, the contents of file 2 will be appended directly to the contents of file 1, and file 2 will then be deleted.

## Appendix B Details of keyword-based and semantic retrieval.

**Table 5: The example of instruction of keyword-based retrieval with single-condition and multi-condition in LSFS and in LLM without LSFS.**
| Task | Task Example | Method | Instruction |
| --- | --- | --- | --- |
| Keyword-based Retrieval<br>(Single-condition) | Find papers in the computer-vision category authored by Emily Zhang. | LLM w/o LSFS | At current step, you need to judge if the input paper satisfy [retrieve condition]. If yes, you should summarize the paper, if no you don’t need to output anything. The paper is [file content] |
|  |  |  |  |
|  |  |  |  |
|  |  | LSFS | LSFS input: Find papers in the computer-vision category authored by Emily Zhang.<br>LLM input: You need to summary the content. The content is [file content] |
| Keyword-based Retrieval<br>(Multi-condition) | Find papers from either Cambridge University or Columbia University. | LLM w/o LSFS | At current step, you need to judge if the input paper satisfy [retrieve condition]. If yes, you should summarize the paper, if no you don’t need to output anything. The paper is [file content] |
|  |  |  |  |
|  |  |  |  |
|  |  | LSFS | LSFS input: Find papers from either Cambridge University or Columbia University.<br>LLM input: You need to summary the content. The content is [file content]. |

**Table 6: The example of instruction of semantic-based retrieval with single-condition and multi-condition in LSFS and in LLM without LSFS.**
| Task | Task Example | Method | Instruction |
| --- | --- | --- | --- |
| Semantic-based Retrieval | Locate the 3 papers showing the highest correlation with reinforcement learning in LLM-training. | LLM w/o LSFS | Fixed prompt: In the next step, you need to accept and remember the paper, but do not generate any outputs. Until you are told to output something.<br>Each input: The paper is [content].<br>After every five entries: Now you can to output the answer. You need to find [retrieve number] papers which most relate to [retrieve condition] from previous record and summary them respectively.<br>Final input: Now you can to output the answer. You need to choose from memory cache to find the [retrieve number] papers that is most relevant to [retrieve condition] |
|  |  |  |  |
|  |  |  |  |
|  |  | LSFS | LSFS input: Locate the 3 papers showing the highest correlation with reinforcement learning in LLM-training.<br>LLM input: You need to summary the content. The content is [file content] |

## Appendix C The Instruction of API

**Table 7: Some examples of instruction of API in Section 4.3. For every API, we provide different instructions in different task condition. The instruction of retrieve-summary API is in Appendix B.**
| API Type | Instruction |
| --- | --- |
| Change-Summary API | LSFS Input:<br>w/ directory: Change the content of /xxxx/xxxx.txt to old-file under llm-directory.<br>w/o directory: Modify /xxxx/xxxx.txt to contain change-file.<br>LLM Input: At current step, you need to summary differences between the two contents, the content before the update is [old file], the content after the update is [new file] |
| Rollback API | LSFS Input:<br>By date: Revert the file named syntax to its version from 2023-6-15.<br>By version number: Rollback the cnn file to the state it was in 3 versions ago. |
| Link API | LSFS Input:<br>w/ period of validity: Provide a link for llm-base that will be active for 3 months.<br>w/o period of validity: Generate a link for system-architecture. |

## Appendix D Experiment Details of File Sharing

In this module, we show some examples that the LLM execution results in a failure to correctly generate shareable links. We give each LLM the following input:You are good at writing code, please write code to generate shared links for the file ’path’.

### D.1 The code cannot generate link

In the experiment, the code generated by LLMs may not produce the correct link or link address. For instance, even after successfully installing the required file packages, the following code block demonstrates that the generated link does not direct to the intended target file. The pseudo-code is in Algorithm 1

Figure: Algorithm 1 Pseudo-code of D.1.

### D.2 The code can only generate local link

In most cases, the generated code will produce links to the corresponding files. The code block typically appears as shown below; however, the links generated by this code are limited to local access and do not provide shareable links for external users. The pseudo-code is in Alg. 2

Figure: Algorithm 2 Pseudo-code of D.2.

### D.3 The code can generate shareable link

In our experiments, the code generated by LLMs can occasionally produce a shareable link. However, generating such a link often involves complex configuration steps. For instance, users need to authorize the Dropbox app, obtain an access token, and perform other setup tasks, as illustrated in the following code block. Moreover, due to the variety of platforms for generating shareable links, LLMs may switch between different platforms with each code generation, leading to considerable user configuration time. The Steps and Pseudo-code in Alg. 3

Figure: Algorithm 3 Procedures of D.3.

In contrast, our Link API simplifies this process: users only need to provide Google Drive credentials, and they can effortlessly generate shareable links without the need for extensive configuration.

## Appendix E Further Discussions

File system is one of the most important components of an operating system. Enhancing and expanding its capability based on LLMs can help to increase its usability for users.
Building on the foundation of LSFS, there are numerous avenues for future research to explore. This section highlights key areas of study that could extend and enhance the core capabilities of LSFS.

#### Multi-modal, Multi-extension File Manipulation

Currently, LSFS primarily supports operations on text files. While it also provides basic functionality for handling other types of files, such as XLSX, JPG, MP3, and MP4, through writing and deletion, it lacks dedicated syscall interfaces tailored to these formats. Future work could focus on leveraging the semantic information embedded in these non-text file types to design and optimize file management operations, enabling more intelligent and context-aware file management across a broader directory of formats.

#### Security and Privacy Enhancements

LSFS must implement robust protection measures to defend against a variety of security threats, such as file data leakage, tampering, and unauthorized access. Additionally, incorporating encryption techniques is essential to secure data interactions and transmissions between LSFS and LLMs, ensuring that file privacy remains protected at all stages of processing and communication.

#### Optimized Retrieval Strategies

The core functionality of LSFS lies in its keyword and semantic retrieval capabilities.
In future work, these retrieval methods can be further optimized by integrating more advanced and precise algorithms, enhancing the overall accuracy and effectiveness of LSFS’s retrieval performance.

#### More Instantiated APIs and syscalls

There are two main directions for future optimization of LSFS design. First, further deepening the integration of traditional file system functions into LSFS. While this paper focuses on the design of the most essential and commonly used syscalls within LSFS, many additional functionalities remain unexplored and can be developed in the future. Second, as user demands for file systems continue to grow, there are numerous operations that traditional file systems are unable to support. To address these needs, LSFS can be expanded to instantiate APIs for related operations, providing users with more versatile and advanced capabilities.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: W1 - W5
soundness: 2
strengths: S1. Semantic file systems enhance file management by incorporating content context, enabling more intuitive and effective operations, which is an important direction.

S2. LSFS simplifies interactions with file system, making file management more accessible and user-friendly.

S3. Integrating LLMs in system-level tasks expands functionality, enabling intelligent, responsive, and user-focused file extraction.
summary: This paper introduces an LLM-based Semantic File System (LSFS), designed to improve file management through natural language prompts, rather than traditional command-based interactions. LSFS integrates large language models (LLMs) to facilitate semantic file operations like retrieval, summarization, and rollback. At its core, LSFS uses a vector database to create semantic indexes for files, enabling high-level file operations that consider the content and context of files. It also includes a comprehensive set of APIs that allow complex operations, such as CRUD, grouping, and semantic retrieval, to be executed through natural language prompts. Experimental results show that LSFS outperforms traditional systems in retrieval accuracy (with a 15% improvement) and speed (2.1x faster), proving especially effective for semantic file tasks that go beyond conventional keyword searches.
weaknesses: W1. The motivation is not concretely convincing, especially the first challenge mentioned in Introduction.

In the Intro section, the authors mentioned that "For instance, if two files have similar content–such as different versions of the same document–traditional file systems lack the capability to organize or retrieve these files based on their content similarity." Why the files should be organized by the similarity of their content? What are the benefits and what are the practical application scenarios? It would be better to at least add one short discussion or a few examples. 

W2. This paper does not point out what key problem they want to solve. Compared to a research paper, it seems more like a technical report.

W3. The experimental setting is questionable. No baselines and introduction of datasets.

W4. The experimental results need more explanation. The traditional file system needs to execute commands to extract files, which should be faster than calling one LLM, even though these LLMs are light-weight. The authors should also list the inference time of LLMs, which should be also counted as the interaction time between users and the file system. Then the authors can also list the time that users manually write commands, which should be a good point to prove the point that -- LSFS can not only speed up file retrieving accuracy and speed, but also can reduce the interaction time between users and file system.

W5. Safety insurance mechanisms is pointed out as one contribution, however, there is no description of this mechanism and no experimental comparison between the performance of LSFS with and without the safety insurance mechanisms.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. Intro (line 57) "The community still lacks a more general LSFS to serve as a common foundation that can be used by various agents on the application-level." Do you have some references backing up this lack? I mean, have people expressed somewhere they'd like/need a FS organized by an intelligent agent, i.e., an LLM here?
2. Related Work (line 133) "Besides, it integrates comprehensive semantic information across all aspects of the file system–from storage and file operations to practical applications" This sentence is a bit vague, could you name some of the semantic information here, so to guide the reader?
3. Could you add a positioning sentence in §2.3 to explain clearly where LSFS delineates from this research axis?
4. In this stage many operations from traditional FS aren't there, from what I understood, this is typically the case for right modification of files or group affiliation… These would be particularly helpful so to "propagate" these rights to any retrievers, preventing typically to see the private/exclusive file of someone else appearing in _my_ search results, wouldn't it?
5. In §4,2 (line 294), the authors mentioned that supervisor updates "periodically" what is the period between each check and therefore how expensive resource-wise is it? Did the authors check various values for this, searching for the sweet-spot between resource-consumption and freshness of the LFSF data? Also how does it scale in terms of file number and disk footprint?
6. Overall, §4.4 seems to be more or less an NL2CSV tool, filling fields of a JSON, right? In such case, this is something that the community has been exploring a lot these past two years, so maybe adding some pointers wouldn't hurt. This goes also for the §5.1 associated with RQ1.
7. Are authors considering releasing their test data for §5.1? Also, it would be good to have some examples in the body of the article.
8. In §5.2 why no QWen or Gemma in the experimental run for Table 2, and no Gemma in Figure 6?
9. In §5.2 still, what about very large number of files?
10. Ibid., same for the number of versions?
soundness: 2
strengths: + Well motivated, especially through the Introduction.
+ Clearly written
+ Very nice figures
+ Hot topic nowadays, with many LLM-based applications reshaping the ways we interact with machines
summary: In this article, the authors based their efforts on the hypothesis that Large language models (LLMs) have the potential to improve file management systems by enabling interactions through natural language rather than traditional manual commands. Following this idea, they proposed LLM-based Semantic File System (LSFS) to address some of the current File System limitations (to the users), by allowing typically semantic file management through natural language prompts. LSFS has through some APIs for semantic file operations, achieves better retrieval accuracy and speed compared to traditional systems or the use of standalone LLMs respectively. It supports complex tasks like semantic file retrieval, rollback, and sharing with high success rates.
weaknesses: ### General Remarks
- Even though the Introduction is clear, I'd have liked a more concrete / detailed example, maybe having a finer-grained figure 1 would help.
- The balance between §3 Vs. §4 is unexpected, I would have imagined a more detailed architecture section (§3), explaining the various design choices. Instead, the authors motivated their architecture. In addition to not being referenced, this motivation -to me- would have been better positioned directly in the Introduction. Similarly, the overview of the architecture and the description of Figure 2 would have benefit also the introduction of an example, especially since Fig2.a doesn't contain precise information, but rather e.g. a list of blocks entitled API.
- Overall, for §5, it would have been very interesting and convincing to see an experiment involving users' usage and performances, using a TFS and the presented LSFS. The authors could have reviewed the success rate and the time efficiency of users in both settings together with collecting feedback from them, following the traditional user studies.
- Very disappointed to have the future directions (as Appendix H) not included at all in the main article, but instead, pushed as optional reading material at the very very end of the .pdf…
- Finally, the overall article, from §3 onwards especially, reads a bit like a technical report and lacks -to me- a step-back to better highlight the novelties and the perspectives while guiding the reading with more examples / intuitions directly in the text.

### Minor Comments:
- Abstract "agent applications **TO** perform file operations"
- Introduction (line 95), "a LLM-based" should be **an**
- Table 1 (line 235), typo on "Hybrid retrieval" better to put everything lower-case as the rest of the table
- In §4.1, in Composite Syscall of LSFS, it would be better if the authors could make explicit the composition of atomic calls, i.e. for each entry, adding a generic formula (or examples) of how the composite call is practically chaining the atomic ones.
- In Figure 4, "Please summary all paper from AAA University about LLM" there's a typo in the second word: **summarize**.
- Similarly, still in Figure 4, "Please use file A update the content of file B" misses the word **to** before 'update'.
- In §5.2, (line 450), typo: "vary the the number of rollback versions" remove one **the**.
- In §5.3, (line 478), "Therefore, we make two enhanced versions, named as TFS-grep and TFS-grep* to make the comparison" I would be great to tell there differences in a line instead of relying on the Appendix, so to make the article (before page 10) self-contained.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Your systems seems to have advantages for searching and retrieving files based on keyword or semantic search. This could be implemented on top of a conventional file system, why implement a new API for that?
2. Is the accuracy of the LSFS parser of about 90% enough for meaningful work? That means 10% incorrect results. How did you collect the NLP commands for this evaluation?
3. How exactly did you perform the evaluation summarized in table 2?
soundness: 2
strengths: - very interesting and original setup
- includes interesting examples how to use the file system
summary: The authors propose, implement and describe an LLM-based semantic file system, where commands are replaced by prompts. They describe APIs and in several experiments compare how this filesystem is used and performs, compared to a conventional file system.
weaknesses: - weak evaluation, based on examples and not on more extensive use cases and human evaluation, evaluation setup not described in detail / cannot be reproduced
- unclear for which users this could be helpful
- unclear how robust the system is when translating NLP input into actual actions
- unclear how the new API maps and extends conventional file handling APIs, and why setting up a new API set is superior to adding some APIs to a conventional file system

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1) Are there guardrails in place to restrict the LLM to not scan through personal data?
2) How much cost are we saving with the new architecture?
3) Are there any security concerns for using this architecture?
4) How much of an operational overhead is this architecture based on traditional architecture ?
5) What are the other use cases of this architecture in real life scenarios?
6) This seems like an ongoing problem that needs to be resolved; are there any similar existing architectures? Have you looked at those papers?
7) Is there an Andon Cord mechanism to stop the LLM to give out hallucinations and wonky results to the user?
8) While scanning through the files, is the data saved in memory? Does the data contain PII information (ppersonal information about the user)?
soundness: 3
strengths: The paper touches on an existing problem that exists in the day-to-day lives of developers and Mac OS users of remembering the file names and directory where the files are present and need modification. There is no way to solve this problem at the present. Even while using LLMs, sometimes developers have to hard code the file path for retrieval. LLM based file retrival system is new and useful for anyone who is fed up of the traditional based systems.They did pretty well work on describing the APIs to be used in the new framework and a commendable job in comparing the APIs to the traditional ones. The quality of the paper was good and the presentation with diagrams were very useful to get the context of the paper.
The architecture of the new framework was explained in detail and they have done a good job in explaining how each component in the architecture is integrated with LLMs. Evaluations are carried out based on success, performance, and performance on non-semantic based tasks like file sharing over sample data/files and are pretty easy to follow.
summary: The paper represents a problem in the current scenario of semantic file matching algorithms; currently, we use the traditional way of semantic matching algorithms based on file name, size, and timestamps. This involved remembering syntax and filenames. This fails in scenarios where two files have similar text; here its hard to distinguish files based on pure string matching. The paper introduces LLM with traditional file systems to do LLM based semantic file management.

LSFS extracts semantic features from the file content and generates corresponding embedding vectors. LSFS incorporates semantic information into its file operations. In Linux, if we need to change a file, i.e., replace a file with another, we need to remember the path, but with LSFS  the users don't need to remember the file name and can talk to LLM to make the changes for them. They have introduced a LLM-based Semantic File System (LSFS), an LSFS parser, and safety insurance mechanisms to the traditional file matching algorithms. The paper has done a great job at explaining the traditional way and modifications done with NLP. They have elaborately explained the API changes they have made over traditional architecture and given diagrams to explain the architecture. Also, they have demonstrated how components of LSFS interact with each other to achieve different functionalities.

Evaluations are carried out based on success, performance, and performance on non-semantic based tasks like file sharing over sample data/files.
weaknesses: The paper could have used an example to walkthrough the implementation. Each component description could have been presented with design diagrams or a flowchart that is easy to understand; visual representation always helps! More evaluations to prove their architecture is better than the traditional ones based on performance, latency, operational burden, and cost. The paper didn't touch on any security concerns while using the LLMS. Are there guardrails in place to restrict the LLMs to scanning through the personal data? One more thing the paper lacked was elaborating on the use cases where this architecture can be used.

Meta-review: The paper proposes LSFS (LLM-based Semantic File System), a novel approach that enhances traditional file systems with semantic understanding through LLMs. It enables natural language interactions for file operations with some designed safety mechanisms. The authors demonstrate LSFS's effectiveness through evaluations on file retrieval and management tasks.

The reviewers value the paper's clear motivation addressing real-world file management challenges, practical architecture design, user studies and expanded experiments with more files and different LLM models. Key concerns discussed include the fairness of user study comparisons between experienced and lay users, security considerations for LLM access to files, parser accuracy and robustness, and limited evaluation on file types beyond basic tasks. 

The authors have engaged constructively with reviewer feedback and provided extensive additional experiments and clarifications. While some concerns about security mechanisms and evaluation comprehensiveness remain to be addressed in future work, I support acceptance at ICLR given the paper's novel contribution to semantic file management and demonstrated practical benefits.

Decision: Accept (Poster)