Repository: Marker-Inc-Korea/AutoRAG

Issue:
[Bug] Restrict 'threshold' config to numbers only for module_type
> Restrict 'threshold' config to numbers only for module_type 

 _Originally posted by @hongsw in [#821](https://github.com/Marker-Inc-Korea/AutoRAG/issues/821#issuecomment-2403716359)_


## Is your feature request related to a problem? Please describe.
Currently, the `threshold` configuration for `module_type` accepts any type of value, which can lead to validation errors and potential runtime issues. Specifically, we've observed instances where date strings (e.g., "2015-01-01T00:00:00.000Z") are being used instead of numerical values. This inconsistency can cause confusion for users and may result in unexpected behavior in our application.

For example, we've seen the following error:

```
TYPE must be number

  216 |             {
  217 |               "module_type": "recency_filter",
> 218 |               "threshold": "2015-01-01T00:00:00.000Z"
      |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^ 👈🏽  type must be number
```

This indicates that a string is being used where a number is expected.

## Describe the solution you'd like
We propose to modify the JSON schema to explicitly restrict the `threshold` configuration for `module_type` to accept only numerical values. This change should:

1. Update the JSON schema to specify that `threshold` must be a number.
2. Provide clear error messages when non-numerical values are used for `threshold`.
3. Update any relevant documentation to clearly state that `threshold` should be a number.

## Describe alternatives you've considered
1. Allow both numbers and date strings, and convert date strings to numerical timestamps internally:
   - Pros: More flexible for users who prefer working with dates
   - Cons: Increases complexity and may lead to inconsistent usage

2. Use a separate configuration for date-based thresholds:
   - Pros: Clear separation between numerical and date-based thresholds
   - Cons: Requires more configuration options, potentially complicating the schema

## Additional context
This change will improve the consistency of our configuration schema and reduce errors caused by incorrect data types. It's part of our ongoing effort to make the configuration process more robust and user-friendly.

Example of the desired schema change:

```json
{
  "properties": {
    "module_type": {
      "type": "object",
      "properties": {
        "threshold": {
          "type": "number"
        }
      }
    }
  }
}
```

This change will ensure that only numerical values are accepted for the `threshold` configuration.

Discussion:
hongsw: Full.yaml file must update. @vkehfdl1 

Patch:
diff --git a/autorag/nodes/passagefilter/base.py b/autorag/nodes/passagefilter/base.py
index d1b4b02..8efe697 100644
--- a/autorag/nodes/passagefilter/base.py
+++ b/autorag/nodes/passagefilter/base.py
@@ -1,14 +1,12 @@
 import abc
-import functools
 import logging
-import os
 from pathlib import Path
-from typing import Union, Tuple, List
+from typing import Union
 
 import pandas as pd
 
 from autorag.schema.base import BaseModule
-from autorag.utils import result_to_dataframe, validate_qa_dataset, fetch_contents
+from autorag.utils import validate_qa_dataset
 
 logger = logging.getLogger("AutoRAG")
 
@@ -50,42 +48,3 @@ class BasePassageFilter(BaseModule, metaclass=abc.ABCMeta):
 		), "previous_result must have retrieved_ids column."
 		ids = previous_result["retrieved_ids"].tolist()
 		return queries, contents, scores, ids
-
-
-# same with passage filter from now
-def passage_filter_node(func):
-	@functools.wraps(func)
-	@result_to_dataframe(["retrieved_contents", "retrieved_ids", "retrieve_scores"])
-	def wrapper(
-		project_dir: Union[str, Path], previous_result: pd.DataFrame, *args, **kwargs
-	) -> Tuple[List[List[str]], List[List[str]], List[List[float]]]:
-		if func.__name__ == "recency_filter":
-			corpus_df = pd.read_parquet(
-				os.path.join(project_dir, "data", "corpus.parquet"), engine="pyarrow"
-			)
-			metadatas = fetch_contents(corpus_df, ids, column_name="metadata")
-			times = [
-				[time["last_modified_datetime"] for time in time_list]
-				for time_list in metadatas
-			]
-			filtered_contents, filtered_ids, filtered_scores = func(
-				contents_list=contents,
-				scores_list=scores,
-				ids_list=ids,
-				time_list=times,
-				*args,
-				**kwargs,
-			)
-		else:
-			filtered_contents, filtered_ids, filtered_scores = func(
-				queries=queries,
-				contents_list=contents,
-				scores_list=scores,
-				ids_list=ids,
-				*args,
-				**kwargs,
-			)
-
-		return filtered_contents, filtered_ids, filtered_scores
-
-	return wrapper
diff --git a/autorag/nodes/passagefilter/recency.py b/autorag/nodes/passagefilter/recency.py
index 5ed9cbe..22b1ba4 100644
--- a/autorag/nodes/passagefilter/recency.py
+++ b/autorag/nodes/passagefilter/recency.py
@@ -35,7 +35,7 @@ class RecencyFilter(BasePassageFilter):
 		scores_list: List[List[float]],
 		ids_list: List[List[str]],
 		time_list: List[List[datetime]],
-		threshold: Union[datetime, date],
+		threshold_datetime: Union[datetime, date],
 	) -> Tuple[List[List[str]], List[List[str]], List[List[float]]]:
 		"""
 		Filter out the contents that are below the threshold datetime.
@@ -46,16 +46,24 @@ class RecencyFilter(BasePassageFilter):
 		:param scores_list: The list of lists of scores retrieved
 		:param ids_list: The list of lists of ids retrieved
 		:param time_list: The list of lists of datetime retrieved
-		:param threshold: The threshold to cut off
+		:param threshold_datetime: The threshold to cut off.
+			In recency filter, you have to use the datetime.datetime object or datetime.date object.
+			All you need to do is to set the date at your YAML file.
+			For example, you can write "2010-09-09 3:45:06" or "2010-09-09" in the YAML file.
 		:return: Tuple of lists containing the filtered contents, ids, and scores
 		"""
-		if not (isinstance(threshold, datetime) or isinstance(threshold, date)):
+		if not (
+			isinstance(threshold_datetime, datetime)
+			or isinstance(threshold_datetime, date)
+		):
 			raise ValueError(
-				f"Threshold should be a datetime object, but got {type(threshold)}"
+				f"Threshold should be a datetime object, but got {type(threshold_datetime)}"
 			)
 
-		if not isinstance(threshold, datetime):
-			threshold = datetime.combine(threshold, datetime.min.time())
+		if not isinstance(threshold_datetime, datetime):
+			threshold_datetime = datetime.combine(
+				threshold_datetime, datetime.min.time()
+			)
 
 		time_list = [
 			list(
@@ -90,7 +98,7 @@ class RecencyFilter(BasePassageFilter):
 				scores_list,
 				ids_list,
 				time_list,
-				[threshold] * len(contents_list),
+				[threshold_datetime] * len(contents_list),
 			)
 		)
 
diff --git a/docs/source/nodes/passage_filter/recency_filter.md b/docs/source/nodes/passage_filter/recency_filter.md
index b59413f..9e33047 100644
--- a/docs/source/nodes/passage_filter/recency_filter.md
+++ b/docs/source/nodes/passage_filter/recency_filter.md
@@ -26,7 +26,7 @@ Plus, the value of the metadata must be `datetime.datetime` object.
   If the time is later than a threshold, the content will be filtered out.
   This is essential to run the module, so you have to set this parameter.
 
-  📌 **threshold** format should be one of the following three!
+  📌 **threshold_datetime** format should be one of the following three!
   - `YYYY-MM-DD`
   - `YYYY-MM-DD HH:MM`
   - `YYYY-MM-DD HH:MM:SS`
@@ -36,5 +36,5 @@ Plus, the value of the metadata must be `datetime.datetime` object.
 ```yaml
 modules:
   - module_type: recency_filter
-    threshold: 2015-01-01
+    threshold_datetime: 2015-01-01
 ```
diff --git a/sample_config/rag/full.yaml b/sample_config/rag/full.yaml
index dbd2443..e65662f 100644
--- a/sample_config/rag/full.yaml
+++ b/sample_config/rag/full.yaml
@@ -88,7 +88,7 @@ node_lines:
         - module_type: similarity_percentile_cutoff
           percentile: 0.6
         - module_type: recency_filter
-          threshold: 2015-01-01
+          threshold_datetime: 2015-01-01 3:45:07
         - module_type: threshold_cutoff
           threshold: 0.85
         - module_type: percentile_cutoff


Test patch:
diff --git a/tests/autorag/nodes/passagefilter/test_recency_filter.py b/tests/autorag/nodes/passagefilter/test_recency_filter.py
index db6660a..de0635c 100644
--- a/tests/autorag/nodes/passagefilter/test_recency_filter.py
+++ b/tests/autorag/nodes/passagefilter/test_recency_filter.py
@@ -26,7 +26,7 @@ def test_recency_filter(recency_filter_instance):
 		scores_example,
 		ids_example,
 		time_list,
-		threshold=date(2021, 6, 30),
+		threshold_datetime=date(2021, 6, 30),
 	)
 	assert id_result[0] == [ids_example[0][1], ids_example[0][2], ids_example[0][3]]
 	assert id_result[1] == [ids_example[1][2], ids_example[1][3]]
@@ -61,7 +61,7 @@ def test_recency_filter_date_time_list(recency_filter_instance):
 		scores_example,
 		ids_example,
 		time_list_2,
-		threshold=date(2021, 6, 30),
+		threshold_datetime=date(2021, 6, 30),
 	)
 	assert id_result[0] == [ids_example[0][1], ids_example[0][2], ids_example[0][3]]
 	assert id_result[1] == [ids_example[1][2], ids_example[1][3]]
@@ -82,7 +82,7 @@ def test_recency_filter_all_filtered(recency_filter_instance):
 		scores_example,
 		ids_example,
 		time_list,
-		threshold=datetime(2040, 6, 30),
+		threshold_datetime=datetime(2040, 6, 30),
 	)
 	assert id_result[0] == [ids_example[0][3]]
 	assert id_result[1] == [ids_example[1][3]]
@@ -99,7 +99,7 @@ def test_recency_filter_minutes(recency_filter_instance):
 		scores_example,
 		ids_example,
 		time_list,
-		threshold=datetime(2022, 3, 5, 12, 35),
+		threshold_datetime=datetime(2022, 3, 5, 12, 35),
 	)
 	assert id_result[0] == [ids_example[0][3]]
 	assert id_result[1] == [ids_example[1][2], ids_example[1][3]]
@@ -116,7 +116,7 @@ def test_recency_filter_seconds(recency_filter_instance):
 		scores_example,
 		ids_example,
 		time_list,
-		threshold=datetime(2022, 3, 5, 12, 40, 30),
+		threshold_datetime=datetime(2022, 3, 5, 12, 40, 30),
 	)
 	assert id_result[0] == [ids_example[0][3]]
 	assert id_result[1] == [ids_example[1][3]]
@@ -131,6 +131,6 @@ def test_recency_filter_node(project_dir_with_corpus):
 	result_df = RecencyFilter.run_evaluator(
 		project_dir=project_dir_with_corpus,
 		previous_result=previous_result,
-		threshold=datetime(2021, 6, 30),
+		threshold_datetime=datetime(2021, 6, 30),
 	)
 	base_passage_filter_node_test(result_df)