Repository: fluent/fluent-bit (C)

Issue:
in_tail: missing log for offset processing due to non-existent old inodes in sqlite.
## Bug Report

**Describe the bug**
When Fluent-bit is monitoring a file that gets deleted after it has been shutdown, and if the inode exists in the SQLite file, it may result in log loss with incorrect offsets from duplicate inodes.

So accurate offset processing is required by comparing the filename of the previous inode.

```
on EXT4

echo "1" > testfile; stat -c %i testfile; rm -f testfile; echo "1" > testfile; stat -c %i testfile
4223667
4223667
```

**To Reproduce**
- Rubular link if applicable:
- Example log message if applicable:
```
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] inode=4486486 with offset=2 appended as /root/testing/bulk2
```
- Steps to reproduce the problem:

```
# clear test files
rm -f /root/testing/bulk1
rm -f /root/testing/bulk2
rm -f /tmp/input.db

# fluentbit start	
/opt/fluent-bit/bin/fluent-bit_2.1.10 -c /opt/fluent-bit/etc/fluent-bit/fluent-bit.conf

echo "1" >> /root/testing/bulk1; stat -c %i /root/testing/bulk1

[2023/10/11 09:51:53] [debug] [input:tail:tail.0] inode=4486486 file=/root/testing/bulk1 promote to TAIL_EVENT
[2023/10/11 09:51:53] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4486486 watch_fd=1 name=/root/testing/bulk1
[2023/10/11 09:51:53] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/10/11 09:51:54] [debug] [task] created task=0x7f5d44a39e60 id=0 OK
[2023/10/11 09:51:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] tail.0: [[1697017913.231894845, {}], {"log"=>"1"}]

# fluentbit stop
# sqlite3 /tmp/input.db
SQLite version 3.7.17 2013-05-20 00:56:22
sqlite> select * from in_tail_files;
1|/root/testing/bulk1|2|4486486|1697017913|0

# file change
rm -f /root/testing/bulk1; echo "2" >> /root/testing/bulk2 ; echo "3" >> /root/testing/bulk2; stat -c %i /root/testing/bulk2

# fluentbit start
# missing "2" log
/opt/fluent-bit/bin/fluent-bit_2.1.10 -c /opt/fluent-bit/etc/fluent-bit/fluent-bit.conf
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] inotify watch fd=30
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] scanning path /root/testing/bulk*
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] inode=4486486 with offset=2 appended as /root/testing/bulk2
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] scan_glob add(): /root/testing/bulk2, inode 4486486
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] 1 new files found on path '/root/testing/bulk*'
[2023/10/11 09:53:11] [ info] [input:storage_backlog:storage_backlog.1] initializing
[2023/10/11 09:53:11] [ info] [input:storage_backlog:storage_backlog.1] storage_strategy='memory' (memory only)
[2023/10/11 09:53:11] [debug] [storage_backlog:storage_backlog.1] created event channels: read=32 write=33
[2023/10/11 09:53:11] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 4.8M
[2023/10/11 09:53:11] [debug] [stdout:stdout.0] created event channels: read=34 write=35
[2023/10/11 09:53:11] [debug] [router] match rule tail.0:stdout.0
[2023/10/11 09:53:11] [debug] [router] match rule storage_backlog.1:stdout.0
[2023/10/11 09:53:11] [ info] [sp] stream processor started
[2023/10/11 09:53:11] [debug] [input chunk] update output instances with new chunk size diff=4096, records=1, input=tail.0
[2023/10/11 09:53:11] [ info] [output:stdout:stdout.0] worker #0 started
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] [static files] processed 2b
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] inode=4486486 file=/root/testing/bulk2 promote to TAIL_EVENT
[2023/10/11 09:53:11] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4486486 watch_fd=1 name=/root/testing/bulk2
[2023/10/11 09:53:11] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/10/11 09:53:11] [debug] [task] created task=0x7f8224639e60 id=0 OK
[2023/10/11 09:53:11] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] tail.0: [[1697017991.013734539, {}], {"log"=>"3"}]

# invalid file name
# size change. 2 => 4
sqlite> select * from in_tail_files;
1|/root/testing/bulk1|4|4486486|1697017913|0

```
**Expected behavior**
Accurate offset processing is required by comparing the filename of the previous inode.

**Screenshots**
None

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used:
  * tested 2.1.10
* Configuration:
```
[SERVICE]
    flush        1
    daemon       Off
    log_level    debug

    parsers_file parsers.conf
    plugins_file plugins.conf
    storage.metrics on
    storage.path /tmp/storage
    storage.sync normal
    storage.checksum off
    storage.backlog.mem_limit 5M

[INPUT]
    Name tail
    Path /root/testing/bulk*
    Exclude_Path *.gz,*.zip

    Read_from_Head true
    Refresh_Interval 3
    Rotate_Wait 10

    storage.type filesystem
    storage.pause_on_chunks_overlimit true

    DB /tmp/input.db
    DB.sync normal
    DB.locking false

[OUTPUT]
    name  stdout
    match *
```
* Environment name and version (e.g. Kubernetes? What version?):
  * docker images CentOS 7.9.2009
  * no k8s
* Server type and version:
  * Intel i5-10400
  * file system : ssd ext4
* Operating System and version:
  * base system : Ubuntu 22.04
* Filters and plugins:
  * in_tail
**Additional context**
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->



Discussion: If there is currently no related PR, I will go ahead and submit a pull request for comparing file names in SQLite.

Please comment on whether I can proceed with the pull request.
I recall some other issues/PRs on this @jinyongchoi - did you have a quick search? However always good to have a PR so please do.
@patrick-stephens There seems to be the expected code in the last comment of a similar issue (#4895 ), but it appears that there is no PR yet. So i will try to submit a PR.
Hello @jinyongchoi,

Thanks for reporting this behavior; I reproduced it by following your instructions.
If you delete a file monitored by Fluent-bit using the tail plugin with the DB option enabled,  you must also delete its entry on the DB to avoid this issue.

Example:
```
$ sqlite3 input.db
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> .tables
in_tail_files
sqlite> select * from in_tail_files;
1|bulk1.log|4|1807178|1697076574|0
sqlite> delete from in_tail_files where id =1 ;
 
sqlite> select * from in_tail_files;
sqlite>
```
Please if the workaround is not an option for you, could you please describe a production use case where you will be deleting files monitored by Fluent-Bit and what the impact of this issue will be for your organization?

Regards.



@RicardoAAD Hi!

In the case of nginx's access.log, it's common to use 'cron scripts' or 'logrotate.'
Assuming you change access.log to access.log-YYYYMMDDHH, following the instructions in this link (https://www.nginx.com/resources/wiki/start/topics/examples/logrotation/),
when this file is rotated and Fluent-Bit is restarted due to modifications in the Fluent-Bit configuration or Lua script, it may forget the inode as shown in the example.
Therefore, manual verification is necessary.

The second scenario pertains to service logs.
Taking the logback module in Spring Boot as an example, it's common to use a format like APP-yyyymmddhh.log. Assuming the usage of such a format, since disk space is finite, log files will be compressed or deleted over time.
In such cases, if a restart is performed for the same reasons mentioned above, the same issue can occur.

While such occurrences are infrequent, when they do happen, the logs are not handled correctly.
Naturally, in certain situations, when the file size is small, it can be recognized as truncated logic and function correctly. Nevertheless, in our operational environment, we've observed cases where inodes from previous timestamps (created field) exist, signifying that this isn't always the situation.

So, to prevent these two scenarios, I plan to test and submit a pull request (PR).
The first change involves deleting items that are not being monitored by Fluent-Bit when it starts.
The second change involves considering the already running Fluent-Bit instances, as files with the same inode will not be deleted.
Therefore, the option being added includes logic to delete files with different file names.

If it doesn't conform to FluentBit's concept, please feel free to inform me.
Thanks!



Fix (patch):
diff --git a/plugins/in_tail/tail.c b/plugins/in_tail/tail.c
index 34a0fec3dbd..52bf2ed6d40 100644
--- a/plugins/in_tail/tail.c
+++ b/plugins/in_tail/tail.c
@@ -372,6 +372,15 @@ static int in_tail_init(struct flb_input_instance *in,
     /* Scan path */
     flb_tail_scan(ctx->path_list, ctx);
 
+#ifdef FLB_HAVE_SQLDB
+    /* Delete stale files that are not monitored from the database */
+    ret = flb_tail_db_stale_file_delete(in, config, ctx);
+    if (ret == -1) {
+        flb_tail_config_destroy(ctx);
+        return -1;
+    }
+#endif
+
     /*
      * After the first scan (on start time), all new files discovered needs to be
      * read from head, so we switch the 'read_from_head' flag to true so any
@@ -725,6 +734,14 @@ static struct flb_config_map config_map[] = {
      "provides higher performance. Note that WAL is not compatible with "
      "shared network file systems."
     },
+    {
+     FLB_CONFIG_MAP_BOOL, "db.compare_filename", "false",
+     0, FLB_TRUE, offsetof(struct flb_tail_config, compare_filename),
+     "This option determines whether to check both the inode and the filename "
+     "when retrieving file information from the db."
+     "'true' verifies both the inode and filename, while 'false' checks only "
+     "the inode (default)."
+    },
 #endif
 
     /* Multiline Options */
diff --git a/plugins/in_tail/tail_config.h b/plugins/in_tail/tail_config.h
index dcfa54e0264..c0263b46503 100644
--- a/plugins/in_tail/tail_config.h
+++ b/plugins/in_tail/tail_config.h
@@ -107,6 +107,7 @@ struct flb_tail_config {
     struct flb_sqldb *db;
     int db_sync;
     int db_locking;
+    int compare_filename;
     flb_sds_t db_journal_mode;
     sqlite3_stmt *stmt_get_file;
     sqlite3_stmt *stmt_insert_file;
diff --git a/plugins/in_tail/tail_db.c b/plugins/in_tail/tail_db.c
index 664963b6dba..6f535ea646b 100644
--- a/plugins/in_tail/tail_db.c
+++ b/plugins/in_tail/tail_db.c
@@ -95,9 +95,38 @@ int flb_tail_db_close(struct flb_sqldb *db)
     return 0;
 }
 
+static int flb_tail_db_file_delete_by_id(struct flb_tail_config *ctx,
+                                         uint64_t id)
+{
+    int ret;
+
+    /* Bind parameters */
+    ret = sqlite3_bind_int64(ctx->stmt_delete_file, 1, id);
+    if (ret != SQLITE_OK) {
+        flb_plg_error(ctx->ins, "db: error binding id=%"PRIu64", ret=%d", id, ret);
+        return -1;
+    }
+
+    ret = sqlite3_step(ctx->stmt_delete_file);
+
+    sqlite3_clear_bindings(ctx->stmt_delete_file);
+    sqlite3_reset(ctx->stmt_delete_file);
+
+    if (ret != SQLITE_DONE) {
+        flb_plg_error(ctx->ins, "db: error deleting stale entry from database:"
+                      " id=%"PRIu64, id);
+        return -1;
+    }
+
+    flb_plg_info(ctx->ins, "db: stale file deleted from database:"
+                 " id=%"PRIu64, id);
+    return 0;
+}
+
 /*
- * Check if an file inode exists in the database. Return FLB_TRUE or
- * FLB_FALSE
+ * Check if an file inode exists in the database.
+ * If the 'compare_filename' option is enabled,
+ * it checks along with the filename. Return FLB_TRUE or FLB_FALSE
  */
 static int db_file_exists(struct flb_tail_file *file,
                           struct flb_tail_config *ctx,
@@ -105,6 +134,7 @@ static int db_file_exists(struct flb_tail_file *file,
 {
     int ret;
     int exists = FLB_FALSE;
+    const unsigned char *name;
 
     /* Bind parameters */
     sqlite3_bind_int64(ctx->stmt_get_file, 1, file->inode);
@@ -116,11 +146,30 @@ static int db_file_exists(struct flb_tail_file *file,
         /* id: column 0 */
         *id = sqlite3_column_int64(ctx->stmt_get_file, 0);
 
+        /* name: column 1 */
+        name = sqlite3_column_text(ctx->stmt_get_file, 1);
+        if (ctx->compare_filename && name == NULL) {
+            flb_plg_error(ctx->ins, "db: error getting name: id=%"PRIu64, *id);
+            return -1;
+        }
+
         /* offset: column 2 */
         *offset = sqlite3_column_int64(ctx->stmt_get_file, 2);
 
         /* inode: column 3 */
         *inode = sqlite3_column_int64(ctx->stmt_get_file, 3);
+
+        /* Checking if the file's name and inode match exactly */
+        if (ctx->compare_filename) {
+            if (flb_tail_target_file_name_cmp((char *) name, file) != 0) {
+                exists = FLB_FALSE;
+                flb_plg_debug(ctx->ins, "db: exists stale file from database:"
+                             " id=%"PRIu64" inode=%"PRIu64" offset=%"PRIu64
+                             " name=%s file_inode=%"PRIu64" file_name=%s",
+                             *id, *inode, *offset, name, file->inode,
+                             file->name);
+            }
+        }
     }
     else if (ret == SQLITE_DONE) {
         /* all good */
@@ -168,6 +217,42 @@ static int db_file_insert(struct flb_tail_file *file, struct flb_tail_config *ct
     return flb_sqldb_last_id(ctx->db);
 }
 
+static int stmt_add_param_concat(struct flb_tail_config *ctx,
+                                 flb_sds_t *stmt_sql, uint64_t count)
+{
+    uint64_t idx;
+    flb_sds_t sds_tmp;
+
+    sds_tmp = flb_sds_cat(*stmt_sql, SQL_STMT_START_PARAM,
+                          SQL_STMT_START_PARAM_LEN);
+    if (sds_tmp == NULL) {
+        flb_plg_debug(ctx->ins, "error concatenating stmt_sql: param start");
+        return -1;
+    }
+    *stmt_sql = sds_tmp;
+
+    for (idx = 1; idx < count; idx++) {
+        sds_tmp = flb_sds_cat(*stmt_sql, SQL_STMT_ADD_PARAM,
+                              SQL_STMT_ADD_PARAM_LEN);
+        if (sds_tmp == NULL) {
+            flb_plg_debug(ctx->ins, "error concatenating stmt_sql: add param");
+            return -1;
+        }
+
+        *stmt_sql = sds_tmp;
+    }
+        
+    sds_tmp = flb_sds_cat(*stmt_sql, SQL_STMT_PARAM_END,
+                          SQL_STMT_PARAM_END_LEN);
+    if (sds_tmp == NULL) {
+        flb_plg_debug(ctx->ins, "error concatenating stmt_sql: param end");
+        return -1;
+    }
+    *stmt_sql = sds_tmp;
+
+    return 0;
+}
+
 int flb_tail_db_file_set(struct flb_tail_file *file,
                          struct flb_tail_config *ctx)
 {
@@ -185,6 +270,11 @@ int flb_tail_db_file_set(struct flb_tail_file *file,
     }
 
     if (ret == FLB_FALSE) {
+        /* Delete stale file of same inode */
+        if (ctx->compare_filename && id > 0) {
+            flb_tail_db_file_delete_by_id(ctx, id);
+        }
+
         /* Get the database ID for this file */
         file->db_id = db_file_insert(file, ctx);
     }
@@ -275,3 +365,128 @@ int flb_tail_db_file_delete(struct flb_tail_file *file,
     flb_plg_debug(ctx->ins, "db: file deleted from database: %s", file->name);
     return 0;
 }
+
+/*
+ * Delete stale file from database
+ */
+int flb_tail_db_stale_file_delete(struct flb_input_instance *ins,
+                                  struct flb_config *config,
+                                  struct flb_tail_config *ctx)
+{
+    int ret = -1;
+    size_t sql_size;
+    uint64_t idx;
+    uint64_t file_count = ctx->files_static_count;
+    flb_sds_t stale_delete_sql;
+    flb_sds_t sds_tmp;
+    sqlite3_stmt *stmt_delete_inodes = NULL;
+    struct mk_list *tmp;
+    struct mk_list *head;
+    struct flb_tail_file *file;
+
+    if (!ctx->db) {
+        return 0;
+    }
+
+    /* Create a stmt sql buffer */
+    sql_size = SQL_DELETE_STALE_FILE_START_LEN;
+    sql_size += SQL_DELETE_STALE_FILE_WHERE_LEN;
+    sql_size += SQL_STMT_START_PARAM_LEN;
+    sql_size += SQL_STMT_PARAM_END_LEN;
+    sql_size += SQL_STMT_END_LEN;
+    if (file_count > 0) {
+        sql_size += (SQL_STMT_ADD_PARAM_LEN * file_count);
+    }
+
+    stale_delete_sql = flb_sds_create_size(sql_size + 1);
+    if (!stale_delete_sql) {
+        flb_plg_error(ctx->ins, "cannot allocate buffer for stale_delete_sql:"
+                      " size: %zu", sql_size);
+        return -1;
+    }
+
+    /* Create a stmt sql */
+    sds_tmp = flb_sds_cat(stale_delete_sql, SQL_DELETE_STALE_FILE_START,
+                          SQL_DELETE_STALE_FILE_START_LEN);
+    if (sds_tmp == NULL) {
+        flb_plg_error(ctx->ins,
+                      "error concatenating stale_delete_sql: start");
+        flb_sds_destroy(stale_delete_sql);
+        return -1;
+    }
+    stale_delete_sql = sds_tmp;
+
+    if (file_count > 0) {
+        sds_tmp = flb_sds_cat(stale_delete_sql, SQL_DELETE_STALE_FILE_WHERE,
+                              SQL_DELETE_STALE_FILE_WHERE_LEN);
+        if (sds_tmp == NULL) {
+            flb_plg_error(ctx->ins,
+                          "error concatenating stale_delete_sql: where");
+            flb_sds_destroy(stale_delete_sql);
+            return -1;
+        }
+        stale_delete_sql = sds_tmp;
+
+        ret = stmt_add_param_concat(ctx, &stale_delete_sql, file_count);
+        if (ret == -1) {
+            flb_plg_error(ctx->ins,
+                          "error concatenating stale_delete_sql: param");
+            flb_sds_destroy(stale_delete_sql);
+            return -1;
+        }
+    }
+
+    sds_tmp = flb_sds_cat(stale_delete_sql, SQL_STMT_END, SQL_STMT_END_LEN);
+    if (sds_tmp == NULL) {
+        flb_plg_error(ctx->ins,
+                      "error concatenating stale_delete_sql: end");
+        flb_sds_destroy(stale_delete_sql);
+        return -1;
+    }
+    stale_delete_sql = sds_tmp;
+
+    /* Prepare stmt */
+    ret = sqlite3_prepare_v2(ctx->db->handler, stale_delete_sql, -1,
+                             &stmt_delete_inodes, 0);
+    if (ret != SQLITE_OK) {
+        flb_plg_error(ctx->ins, "error preparing database SQL statement:"
+                      " stmt_delete_inodes sql:%s, ret=%d", stale_delete_sql,
+                      ret);
+        flb_sds_destroy(stale_delete_sql);
+        return -1;
+    }
+
+    /* Bind parameters */
+    idx = 1;
+    mk_list_foreach_safe(head, tmp, &ctx->files_static) {
+        file = mk_list_entry(head, struct flb_tail_file, _head);
+        ret = sqlite3_bind_int64(stmt_delete_inodes, idx, file->inode);
+        if (ret != SQLITE_OK) {
+            flb_plg_error(ctx->ins, "error binding to stmt_delete_inodes:"
+                          " inode=%lu, ret=%d", file->inode, ret);
+            sqlite3_finalize(stmt_delete_inodes);
+            flb_sds_destroy(stale_delete_sql);
+            return -1;
+        }
+        idx++;
+    }
+
+    /* Run the delete inodes */
+    ret = sqlite3_step(stmt_delete_inodes);
+    if (ret != SQLITE_DONE) {
+        sqlite3_finalize(stmt_delete_inodes);
+        flb_sds_destroy(stale_delete_sql);
+        flb_plg_error(ctx->ins, "cannot execute delete stale inodes: ret=%d",
+                      ret);
+        return -1;
+    }
+
+    ret = sqlite3_changes(ctx->db->handler);
+    flb_plg_info(ctx->ins, "db: delete unmonitored stale inodes from the"
+                 " database: count=%d", ret);
+
+    sqlite3_finalize(stmt_delete_inodes);
+    flb_sds_destroy(stale_delete_sql);
+
+    return 0;
+}
diff --git a/plugins/in_tail/tail_db.h b/plugins/in_tail/tail_db.h
index 7b5355d229c..b1fde721d29 100644
--- a/plugins/in_tail/tail_db.h
+++ b/plugins/in_tail/tail_db.h
@@ -40,4 +40,7 @@ int flb_tail_db_file_rotate(const char *new_name,
                             struct flb_tail_config *ctx);
 int flb_tail_db_file_delete(struct flb_tail_file *file,
                             struct flb_tail_config *ctx);
+int flb_tail_db_stale_file_delete(struct flb_input_instance *ins,
+                                  struct flb_config *config,
+                                  struct flb_tail_config *ctx);
 #endif
diff --git a/plugins/in_tail/tail_sql.h b/plugins/in_tail/tail_sql.h
index 855933a0149..bf724f318cd 100644
--- a/plugins/in_tail/tail_sql.h
+++ b/plugins/in_tail/tail_sql.h
@@ -53,6 +53,28 @@
 #define SQL_DELETE_FILE                                                 \
     "DELETE FROM in_tail_files WHERE id=@id;"
 
+#define SQL_STMT_START_PARAM "(?"
+#define SQL_STMT_START_PARAM_LEN (sizeof(SQL_STMT_START_PARAM) - 1)
+
+#define SQL_STMT_ADD_PARAM ",?"
+#define SQL_STMT_ADD_PARAM_LEN (sizeof(SQL_STMT_ADD_PARAM) - 1)
+
+#define SQL_STMT_PARAM_END ")"
+#define SQL_STMT_PARAM_END_LEN (sizeof(SQL_STMT_PARAM_END) - 1)
+
+#define SQL_STMT_END ";"
+#define SQL_STMT_END_LEN (sizeof(SQL_STMT_END) - 1)
+
+#define SQL_DELETE_STALE_FILE_START                                     \
+    "DELETE FROM in_tail_files "
+#define SQL_DELETE_STALE_FILE_START_LEN                                 \
+    (sizeof(SQL_DELETE_STALE_FILE_START) - 1)
+
+#define SQL_DELETE_STALE_FILE_WHERE                                     \
+    "WHERE inode NOT IN "
+#define SQL_DELETE_STALE_FILE_WHERE_LEN                                 \
+    (sizeof(SQL_DELETE_STALE_FILE_WHERE) - 1)
+
 #define SQL_PRAGMA_SYNC                         \
     "PRAGMA synchronous=%i;"