Repository: plabayo/venndb (language: rust)

Issue:
allow for a row to be validated with custom logic prior to appending it (in any way)
Rust's type system helps a lot with validation, given plenty of assumptions and requirements can be registered within the used types. At times however the combination of properties give meaning as a group, where some combinations make sense and others not. Enums (sum types) are what help usually in this situation. However it is not always a desired one, and in the case of venndb it doesn't play very well with the rest of the setup. As such for that reason and others it can be useful to allow for a custom validation of a row prior to appending it, such that one can guarantee that any appended rows do fulfill all required requirements.

Example:

```rust
#[derive(Debug, VennDB)]
#[venndb(name = "MyDB", validatator = "my_validator_fn")]
pub struct Value {
   pub foo: String,
   pub bar: u32,
}

fn my_validator_fn(value: &Value) -> bool {
    !foo.is_empty() && value.bar > 0
}

let mut db = MyDB::default();
assert!(db.append(Value {
    foo: "".to_owned(),
    bar: 42,
}).is_err()); // fails because foo == empty
```

The above example illustrates that the type system can only bring you so far.
Of course one can argue that a NewType can solve this particular issue by only allowing the newtype
instances to be created with values that match such conditions. And this is true.

However, what if the legal value space of a column depends upon the specific value of another?
E.g. what is `foo` can be empty when `bar` is 2. This becomes a lot more difficult, and given the
large space that unsigned integers are composed from it might also become impossible
to  abstract such rules in a sum type. If desired at all of course, given it would also make
the usage of it within a venndb-derived db more difficult.
any filter values should only get row with any values back
The behaviour where inserted rows with an any value for a focussed property that match for all given values of that property is correct. This is desired and what is wanted. It is the opposite direction however which is currently wrong and which will require another breaking change release.

When querying one can specify a value desired for a filter map property. This can also be an any value given it lives within the same type dimension as the property of that row to begin with. At first this was seen as a side effect and not desired. However this is actually meaningful and in that light it turns out that current behaviour is wrong. In v0.3 (and before) we ignore the query value in case it is an any value, which has the indirect effect that all rows will match regardless of the value. This also includes undefined ones. This is however incorrect for two reasons:

- it does not transfer the right intent. An any value has a specific meaning, that it can be consumed as anything. This is very different from a row with a specific value. When a specific value is searched it is correct for a row with an any value for that property to be matched among the rows with the specific values. However it is not correct for rows with specific values to match when searching for rows with an any value. Querying for an instance is only the beginning. The reason why a row was desired to begin with changes the game entirely;
- matching on rows with specific values, when searching for rows with any value, also rigs the odds on what value to eventually land. Any value is perfectly fair as it can equally match on any of the possible values within the type's dimension space. As soon as you add a specific value to that mix you however have a greater chance to land on that specific value over all the other possibilities. The greater the dimension space the more the game is rigged.

This has to be corrected, but does require a breaking change, as it is a change in behaviour correction.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 719e238..f3d6c70 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,9 +5,44 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+# 0.4.0 (2024-04-19)
+
+Breaking Changes:
+
+* [[`#7`](https://github.com/plabayo/venndb/issues/7)]: correct the behaviour of any filter map query values:
+  - When using an any value as a query filter map value it will now only match rows
+    which have an any value registered for the row;
+  - Prior to this release it was matching on all rows, as if the filter wasn't defined.
+    This seemed correct when deciding on it, but on hindsight is is incorrect behaviour.
+
+New Features:
+
+* [[`#8`](https://github.com/plabayo/venndb/issues/8)]: support custom validations of rows prior to appending them
+
+Example:
+
+```rust
+#[derive(Debug, VennDB)]
+#[venndb(name = "MyDB", validator = my_validator_fn)]
+pub struct Value {
+   pub foo: String,
+   pub bar: u32,
+}
+
+fn my_validator_fn(value: &Value) -> bool {
+    !value.foo.is_empty() && value.bar > 0
+}
+
+let mut db = MyDB::default();
+assert!(db.append(Value {
+    foo: "".to_owned(),
+    bar: 42,
+}).is_err()); // fails because foo == empty
+```
+
 # 0.3.0 (2024-04-18)
 
-Breaking changes:
+Breaking Changes:
 
 * [[`#6`](https://github.com/plabayo/venndb/issues/6)] query filter maps now accept arguments as `impl Into<T>` instead of `T`,
   this can be a breaking change for users that were inserting them as `value.into()`,
diff --git a/Cargo.toml b/Cargo.toml
index 3852b2b..c6dfadd 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -12,7 +12,7 @@ repository = "https://github.com/plabayo/venndb"
 keywords = ["database", "db", "memory", "bits"]
 categories = ["database"]
 authors = ["Glen De Cauwsemaecker <glen@plabayo.tech>"]
-version = "0.3.0"
+version = "0.4.0"
 rust-version = "1.75.0"
 
 [package.metadata.docs.rs]
@@ -23,7 +23,7 @@ rustdoc-args = ["--cfg", "docsrs"]
 bitvec = "1.0.1"
 hashbrown = "0.14.3"
 rand = "0.8.5"
-venndb-macros = { version = "0.3.0", path = "venndb-macros" }
+venndb-macros = { version = "0.4.0", path = "venndb-macros" }
 
 [dev-dependencies]
 divan = "0.1.14"
diff --git a/README.md b/README.md
index 093ad2a..5c2b69a 100644
--- a/README.md
+++ b/README.md
@@ -304,6 +304,89 @@ let hr_employees: Vec<_> = query.department(Department::Hr).execute().unwrap().i
 assert_eq!(hr_employees.len(), 2);
 ```
 
+> ❓ How can I provide custom validation of rows prior to them getting appended?
+
+Is is possible to validate a row based on one or multiple of its properties? Validate in function of relationship
+between multiple properties? Is it possible to provide custom validation to prevent rows
+from getting appended that do not adhere to custom validation rules?
+
+Yes to all of the above.
+
+Example:
+
+```rust,ignore
+#[derive(Debug, VennDB)]
+#[venndb(validator = my_validator_fn)]
+pub struct Value {
+   pub foo: String,
+   pub bar: u32,
+}
+
+fn my_validator_fn(value: &Value) -> bool {
+    !value.foo.is_empty() && value.bar > 0
+}
+
+let mut db = ValueDB::default();
+assert!(db.append(Value {
+    foo: "".to_owned(),
+    bar: 42,
+}).is_err()); // fails because foo == empty
+```
+
+> ❓ Why do `any` filter values only match rows that have an `any` value for that property?
+
+Let's say I have the following `struct`:
+
+```rust,ignore
+use venndb::{Any, VennDB};
+
+#[derive(Debug, VennDB)]
+pub struct Value {
+   #[venndb(filter, any)]
+   pub foo: MyString,
+   pub bar: u32,
+}
+
+#[derive(Debug)]
+pub struct MyString(String);
+
+impl Any for MyString {
+    fn is_any(&self) -> bool {
+        self.0 == "*"
+    }
+}
+
+let db = ValueDB::from_iter([
+    Value {
+        foo: MyString("foo".to_owned()),
+        bar: 8,
+    },
+    Value {
+        foo: MyString("*".to_owned()),
+        bar: 16,
+    }
+].into_Iter()).unwrap();
+
+let mut query = db.query();
+query.foo(MyString("*".to_owned()));
+let value = query.execute().unwrap().any();
+// this will never match the row with bar == 8,
+// tiven foo != an any value
+assert_eq!(value.bar, 16);
+```
+
+Why is this true? Because it is correct.
+
+Allowing it also to match the value `foo` would unfairly
+give more chances for `foo` to be selected over the _any_ value.
+This might not seem like a big difference, but it is. Because what if
+we generate a random string for `Value`s with an _any value? If we
+would allow all rows to be matched then that logic is now rigged,
+with a value of `foo` being more likely then other strings.
+
+As such the only correct answer when filtering for _any_ value,
+is to return rows that have _any_ value.
+
 ## Example
 
 Here follows an example demonstrating all the features of `VennDB`.
@@ -318,7 +401,7 @@ use venndb::VennDB;
 #[derive(Debug, VennDB)]
 // These attributes are optional,
 // e.g. by default the database would be called `EmployeeDB` (name + 'DB').
-#[venndb(name = "EmployeeInMemDB")]
+#[venndb(name = "EmployeeInMemDB", validator = employee_validator)]
 pub struct Employee {
     // you can use the `key` arg to be able to get an `Employee` instance
     // directly by this key. It will effectively establishing a mapping from key to a reference
@@ -363,6 +446,10 @@ pub struct Employee {
     country: Option<String>,
 }
 
+fn employee_validator(employee: &Employee) -> bool {
+    employee.id > 0
+}
+
 fn main() {
     let db = EmployeeInMemDB::from_iter([
         RawCsvRow("1,John Doe,true,false,true,false,Engineering,USA"),
@@ -471,15 +558,21 @@ fn main() {
     assert_eq!(usa_employees[0].id, 1);
 
     println!(">>> At any time you can also append new employees to the DB...");
-    assert!(db
+    assert_eq!(EmployeeInMemDBErrorKind::DuplicateKey, db
         .append(RawCsvRow("8,John Doe,true,false,true,false,Engineering,"))
-        .is_err());
+        .unwrap_err().kind());
     println!(">>> This will fail however if a property is not correct (e.g. ID (key) is not unique in this case), let's try this again...");
     assert!(db
         .append(RawCsvRow("9,John Doe,false,true,true,false,Engineering,"))
         .is_ok());
     assert_eq!(db.len(), 9);
 
+    println!(">>> Rows are also validated prior to appending in case a validator is defined...");
+    println!("    The next insertion will fail due to the id being zero, a condition defined in the custom validator...");
+    assert_eq!(EmployeeInMemDBErrorKind::InvalidRow, db
+        .append(RawCsvRow("0,John Doe,true,false,true,false,Engineering,"))
+        .unwrap_err().kind());
+
     println!(">>> This new employee can now also be queried for...");
     let mut query = db.query();
     query.department(Department::Engineering).is_manager(false);
@@ -607,7 +700,7 @@ In this chapter we'll list the API as generated by `VennDB` for the following ex
 
 ```rust,ignore
 #[derive(Debug, VennDB)]
-#[venndb(name = "EmployeeInMemDB")]
+#[venndb(name = "EmployeeInMemDB", validator = employee_validator)]
 pub struct Employee {
     #[venndb(key)]
     id: u32,
@@ -651,7 +744,7 @@ Database: (e.g. `EmployeeInMemDB`):
 | `EmployeeInMemDB::from_rows(rows: ::std::vec::Vec<Employee>) -> EmployeeInMemDB` or `EmployeeInMemDB::from_rows(rows: ::std::vec::Vec<Employee>) -> Result<EmployeeInMemDB, EmployeeInMemDBError<::std::vec::Vec<Employee>>>` | constructor to create the database directly from a heap-allocated list of data instances. The second version is the one used if at least one `#[venndb(key)]` property is defined, otherwise it is the first one (without the `Result`). |
 | `EmployeeInMemDB::from_iter(iter: impl ::std::iter::IntoIterator<Item = impl ::std::convert::Into<Employee>>) -> EmployeeInMemDB` or `EmployeeInMemDB::from_rows(iter: impl ::std::iter::IntoIterator<Item = impl ::std::convert::Into<Employee>>) -> Result<EmployeeInMemDB, EmployeeInMemDBError<::std::vec::Vec<Employee>>>` | Same as `from_rows` but using an iterator instead. The items do not have to be an `Employee` but can be anything that can be turned into one. E.g. in our example above we defined a struct `RawCsvRow` that was turned on the fly into an `Employee`. This happens all at once prior to inserting the database, which is why the version with a result does return a `Vec` and not an iterator. |
 | `EmployeeInMemDB::append(&mut self, data: impl ::std::convert::Into<Employee>)` or `EmployeeInMemDB::append(&mut self, data: impl ::std::convert::Into<Employee>) -> Result<(), EmployeeInMemDBError<Employee>>` | append a single row to the database. Depending on whether or not a `#[venndb(key)]` property is defined it will generate the `Result` version or not. Same as `from_rows` and `from_iter` |
-| `EmployeeInMemDB::extend<I, Item>(&mut self, iter: I) where I: ::std::iter::IntoIterator<Item = Item>, Item: ::std::convert::Into<Employee>` or `EmployeeInMemDB::extend<I, Item>(&mut self, iter: I) -> Result<(), EmployeeInMemDBError<(Employee, I::IntoIter)>> where I: ::std::iter::IntoIterator<Item = Item>, Item: ::std::convert::Into<Employee>` | extend the database with the given iterator, once again returning a result in case such insertion can go wrong (e.g. because keys are used (duplication)). Otherwise this function will return nothing. |
+| `EmployeeInMemDB::extend<I, Item>(&mut self, iter: I) where I: ::std::iter::IntoIterator<Item = Item>, Item: ::std::convert::Into<Employee>` or `EmployeeInMemDB::extend<I, Item>(&mut self, iter: I) -> Result<(), EmployeeInMemDBError<(Employee, I::IntoIter)>> where I: ::std::iter::IntoIterator<Item = Item>, Item: ::std::convert::Into<Employee>` | extend the database with the given iterator, once again returning a result in case such insertion can go wrong (e.g. because keys are used (duplication) or a row is invalid in case a validator is defined). Otherwise this function will return nothing. |
 | `EmployeeInMemDB::get_by_id<Q>(&self, data: impl ::std::convert::Into<Employee>) -> Option<&Employee> where Employee ::std::borrow::Borrow<Q>, Q: ::std::hash::Hash + ::std::cmp::Eq + ?::std::marker::Sized` | look up a row by the `id` key property. This method will be generated for each property marked with `#[venndb(key)`. e.g. if you have key property named `foo: MyType` property there will be also a `get_by_foo(&self, ...)` method generated. |
 | `EmployeeInMemDB::query(&self) -> EmployeeInMemDBQuery` | create a `EmployeeInMemDBQuery` builder to compose a filter composition to query the database. The default builder will match all rows. See the method API for `EmployeeInMemDBQuery` for more information |
 
diff --git a/venndb-macros/Cargo.toml b/venndb-macros/Cargo.toml
index 79b766f..189514f 100644
--- a/venndb-macros/Cargo.toml
+++ b/venndb-macros/Cargo.toml
@@ -9,7 +9,7 @@ repository = "https://github.com/plabayo/venndb"
 keywords = ["database", "db", "memory", "bits"]
 categories = ["database", "db"]
 authors = ["Glen De Cauwsemaecker <glen@plabayo.tech>"]
-version = "0.3.0"
+version = "0.4.0"
 rust-version = "1.75.0"
 
 [package.metadata.docs.rs]
diff --git a/venndb-macros/src/errors.rs b/venndb-macros/src/errors.rs
index 08d1dab..f63b5b1 100644
--- a/venndb-macros/src/errors.rs
+++ b/venndb-macros/src/errors.rs
@@ -62,6 +62,15 @@ impl Errors {
         ),
     ];
 
+    pub fn expect_path<'a>(&self, e: &'a syn::Expr) -> Option<&'a syn::Path> {
+        if let syn::Expr::Path(path) = e {
+            Some(&path.path)
+        } else {
+            self.unexpected_value("path", e);
+            None
+        }
+    }
+
     fn unexpected_lit(&self, expected: &str, found: &syn::Expr) {
         fn lit_kind(lit: &syn::Lit) -> &'static str {
             use syn::Lit::{Bool, Byte, ByteStr, Char, Float, Int, Str, Verbatim};
@@ -126,6 +135,73 @@ impl Errors {
         )
     }
 
+    fn unexpected_value(&self, expected: &str, found: &syn::Expr) {
+        fn expr_kind(expr: &syn::Expr) -> &'static str {
+            use syn::Expr::{
+                Array, Assign, Async, Await, Binary, Block, Break, Call, Cast, Closure, Const,
+                Continue, Field, ForLoop, Group, If, Index, Infer, Let, Lit, Loop, Macro, Match,
+                MethodCall, Paren, Path, Range, Reference, Repeat, Return, Struct, Try, TryBlock,
+                Tuple, Unary, Unsafe, Verbatim, While, Yield,
+            };
+            match expr {
+                Array(_) => "array",
+                Assign(_) => "assignment",
+                Async(_) => "async block",
+                Await(_) => "await",
+                Binary(_) => "binary operation",
+                Block(_) => "block",
+                Break(_) => "break",
+                Call(_) => "function call",
+                Cast(_) => "cast",
+                Closure(_) => "closure",
+                Const(_) => "const",
+                Continue(_) => "continue",
+                Field(_) => "field access",
+                ForLoop(_) => "for loop",
+                Group(_) => "group",
+                If(_) => "if",
+                Index(_) => "index",
+                Infer(_) => "inferred type",
+                Let(_) => "let",
+                Lit(_) => "literal",
+                Loop(_) => "loop",
+                Macro(_) => "macro",
+                Match(_) => "match",
+                MethodCall(_) => "method call",
+                Paren(_) => "parentheses",
+                Path(_) => "path",
+                Range(_) => "range",
+                Reference(_) => "reference",
+                Repeat(_) => "repeat",
+                Return(_) => "return",
+                Struct(_) => "struct",
+                Try(_) => "try",
+                TryBlock(_) => "try block",
+                Tuple(_) => "tuple",
+                Unary(_) => "unary operation",
+                Unsafe(_) => "unsafe block",
+                Verbatim(_) => "verbatim",
+                While(_) => "while",
+                Yield(_) => "yield",
+                _ => "unknown expression kind",
+            }
+        }
+
+        self.err(
+            found,
+            &[
+                "Expected ",
+                expected,
+                " attribute, found ",
+                found.to_token_stream().to_string().as_str(),
+                " attribute (",
+                expr_kind(found),
+                ")",
+            ]
+            .concat(),
+        )
+    }
+
     /// Issue an error relating to a particular `Spanned` structure.
     pub fn err(&self, spanned: &impl syn::spanned::Spanned, msg: &str) {
         self.err_span(spanned.span(), msg);
diff --git a/venndb-macros/src/generate_db.rs b/venndb-macros/src/generate_db.rs
index 7ee35e4..ecc8b00 100644
--- a/venndb-macros/src/generate_db.rs
+++ b/venndb-macros/src/generate_db.rs
@@ -1,21 +1,23 @@
 use crate::field::{FieldInfo, StructField};
 use proc_macro2::TokenStream;
 use quote::{format_ident, quote, ToTokens};
-use syn::Ident;
+use syn::{Ident, Path};
 
 /// Generate the venndb logic
 pub fn generate_db(
     name: &Ident,
     name_db: &Ident,
+    validator: Option<&Path>,
     vis: &syn::Visibility,
     fields: &[StructField],
 ) -> TokenStream {
     let fields: Vec<_> = fields.iter().filter_map(StructField::info).collect();
 
-    let db_error = DbError::new(&fields[..]);
+    let db_error = DbError::new(validator, &fields[..]);
 
     let db_struct = generate_db_struct(name, name_db, vis, &fields[..]);
-    let db_struct_methods = generate_db_struct_methods(name, name_db, vis, &db_error, &fields[..]);
+    let db_struct_methods =
+        generate_db_struct_methods(name, name_db, validator, vis, &db_error, &fields[..]);
 
     let db_query = generate_query_struct(name, name_db, vis, &fields[..]);
 
@@ -92,6 +94,7 @@ fn generate_db_struct(
 fn generate_db_struct_methods(
     name: &Ident,
     name_db: &Ident,
+    validator: Option<&Path>,
     vis: &syn::Visibility,
     db_error: &DbError,
     fields: &[FieldInfo],
@@ -101,7 +104,8 @@ fn generate_db_struct_methods(
     let method_from_rows =
         generate_db_struct_method_from_rows(name, name_db, vis, db_error, fields);
     let field_methods = generate_db_struct_field_methods(name, name_db, vis, fields);
-    let method_append = generate_db_struct_method_append(name, name_db, vis, db_error, fields);
+    let method_append =
+        generate_db_struct_method_append(name, name_db, validator, vis, db_error, fields);
 
     quote! {
         #[allow(clippy::unused_unit)]
@@ -315,6 +319,7 @@ fn generate_db_struct_method_from_rows(
 fn generate_db_struct_method_append(
     name: &Ident,
     name_db: &Ident,
+    validator: Option<&Path>,
     vis: &syn::Visibility,
     db_error: &DbError,
     fields: &[FieldInfo],
@@ -325,6 +330,18 @@ fn generate_db_struct_method_append(
         name
     );
 
+    let validator_check = match validator {
+        Some(validator) => {
+            let err = DbError::generate_invalid_row_error_kind_creation(name_db);
+            quote! {
+                if !#validator(&data) {
+                    return Err(#err);
+                }
+            }
+        }
+        None => quote! {},
+    };
+
     let db_field_insert_checks: Vec<_> = fields
         .iter()
         .filter_map(|info| match info {
@@ -538,6 +555,7 @@ fn generate_db_struct_method_append(
         }
 
         fn append_internal(&mut self, data: &#name, index: usize) -> #append_kind_return_type {
+            #validator_check
             #(#db_field_insert_checks)*
             #(#db_field_insert_commits)*
             #append_return_output
@@ -759,7 +777,9 @@ fn generate_query_struct_impl(
                 let filter_vec_name: Ident = field.filter_vec_name();
                 let value_filter = match field.filter_any_name() {
                     Some(filter_any_vec) => quote! {
-                        if !::venndb::Any::is_any(&value) {
+                        if ::venndb::Any::is_any(&value) {
+                            filter &= &self.db.#filter_any_vec;
+                        } else {
                             match self.db.#filter_map_name.get(value) {
                                 Some(index) => filter &= &self.db.#filter_vec_name[*index],
                                 None => filter &= &self.db.#filter_any_vec,
@@ -954,6 +974,7 @@ struct DbError {
 #[derive(Debug)]
 enum DbErrorKind {
     DuplicateKey,
+    InvalidRow,
 }
 
 impl ToTokens for DbErrorKind {
@@ -965,18 +986,27 @@ impl ToTokens for DbErrorKind {
                     DuplicateKey,
                 });
             }
+            Self::InvalidRow => {
+                tokens.extend(quote! {
+                    /// The error kind for when the row to be inserted is invalid.
+                    InvalidRow,
+                });
+            }
         }
     }
 }
 
 impl DbError {
-    fn new(fields: &[FieldInfo]) -> Self {
-        let error_duplicate_key = fields.iter().any(|info| matches!(info, FieldInfo::Key(_)));
-        let error_kinds = if error_duplicate_key {
-            vec![DbErrorKind::DuplicateKey]
-        } else {
-            Vec::new()
-        };
+    fn new(validator: Option<&Path>, fields: &[FieldInfo]) -> Self {
+        let mut error_kinds = Vec::new();
+
+        if validator.is_some() {
+            error_kinds.push(DbErrorKind::InvalidRow);
+        }
+
+        if fields.iter().any(|info| matches!(info, FieldInfo::Key(_))) {
+            error_kinds.push(DbErrorKind::DuplicateKey);
+        }
 
         Self { error_kinds }
     }
@@ -988,6 +1018,13 @@ impl DbError {
         }
     }
 
+    fn generate_invalid_row_error_kind_creation(name_db: &Ident) -> TokenStream {
+        let ident_error_kind = format_ident!("{}ErrorKind", name_db);
+        quote! {
+            #ident_error_kind::InvalidRow
+        }
+    }
+
     fn generate_fn_error_kind_usage(
         &self,
         name_db: &Ident,
diff --git a/venndb-macros/src/lib.rs b/venndb-macros/src/lib.rs
index 6fa993e..3b0886c 100644
--- a/venndb-macros/src/lib.rs
+++ b/venndb-macros/src/lib.rs
@@ -92,7 +92,13 @@ fn impl_from_args_struct(
         None => format_ident!("{}DB", name),
     };
 
-    let db_code = generate_db::generate_db(name, &name_db, vis, &fields[..]);
+    let db_code = generate_db::generate_db(
+        name,
+        &name_db,
+        type_attrs.validator.as_ref(),
+        vis,
+        &fields[..],
+    );
 
     quote! {
         #db_code
diff --git a/venndb-macros/src/parse_attrs.rs b/venndb-macros/src/parse_attrs.rs
index fe6986c..480fea5 100644
--- a/venndb-macros/src/parse_attrs.rs
+++ b/venndb-macros/src/parse_attrs.rs
@@ -157,6 +157,7 @@ fn is_bool(ty: &syn::Type) -> bool {
 #[derive(Default)]
 pub struct TypeAttrs {
     pub name: Option<syn::LitStr>,
+    pub validator: Option<syn::Path>,
 }
 
 impl TypeAttrs {
@@ -177,6 +178,10 @@ impl TypeAttrs {
                     if let Some(m) = errors.expect_meta_name_value(&meta) {
                         this.name = errors.expect_lit_str(&m.value).cloned();
                     }
+                } else if name.is_ident("validator") {
+                    if let Some(m) = errors.expect_meta_name_value(&meta) {
+                        this.validator = errors.expect_path(&m.value).cloned();
+                    }
                 } else {
                     errors.err(
                         &meta,
diff --git a/venndb-usage/src/main.rs b/venndb-usage/src/main.rs
index 7b0892d..af5d1ea 100644
--- a/venndb-usage/src/main.rs
+++ b/venndb-usage/src/main.rs
@@ -3,6 +3,7 @@
 use venndb::{Any, VennDB};
 
 #[derive(Debug, VennDB)]
+#[venndb(validator = employee_validator)]
 pub struct Employee {
     #[venndb(key)]
     id: u32,
@@ -15,6 +16,10 @@ pub struct Employee {
     department: Department,
 }
 
+fn employee_validator(employee: &Employee) -> bool {
+    employee.id > 0
+}
+
 #[derive(Debug)]
 pub struct L1Engineer {
     id: u32,
@@ -64,7 +69,7 @@ fn main() {
 }
 
 #[cfg(test)]
-mod tests {
+mod tests_v0_1 {
     use super::*;
 
     #[test]
@@ -868,8 +873,77 @@ mod tests_v0_2_1 {
         department: Option<Department>,
     }
 
+    // these two tests are no longer correct since
+    // the fix introduced in issue https://github.com/plabayo/venndb/issues/7
+    //
+    // this is intended. As such these issues have moved to `::tests_v0_4`.
+    // Check out the above issue if you want to find the motivation why.
+
+    // #[test]
+    // fn test_any_filter_map() {
+    //     let db = EmployeeDB::from_rows(vec![
+    //         Employee {
+    //             id: 1,
+    //             name: "Alice".to_string(),
+    //             is_manager: true,
+    //             is_admin: false,
+    //             is_active: true,
+    //             department: Department::Engineering,
+    //         },
+    //         Employee {
+    //             id: 2,
+    //             name: "Bob".to_string(),
+    //             is_manager: false,
+    //             is_admin: false,
+    //             is_active: true,
+    //             department: Department::HR,
+    //         },
+    //     ])
+    //     .unwrap();
+
+    //     let mut query = db.query();
+    //     query.department(Department::Any);
+    //     let results = query.execute().unwrap().iter().collect::<Vec<_>>();
+    //     assert_eq!(results.len(), 2);
+    //     assert_eq!(results[0].id, 1);
+    //     assert_eq!(results[1].id, 2);
+    // }
+
+    // #[test]
+    // fn test_any_option_filter_map() {
+    //     let db = WorkerDB::from_rows(vec![
+    //         Worker {
+    //             id: 1,
+    //             is_admin: false,
+    //             is_active: Some(true),
+    //             department: Some(Department::Engineering),
+    //         },
+    //         Worker {
+    //             id: 2,
+    //             is_admin: false,
+    //             is_active: Some(true),
+    //             department: Some(Department::HR),
+    //         },
+    //         Worker {
+    //             id: 3,
+    //             is_admin: false,
+    //             is_active: None,
+    //             department: None,
+    //         },
+    //     ])
+    //     .unwrap();
+
+    //     let mut query = db.query();
+    //     query.department(Department::Any);
+    //     let results = query.execute().unwrap().iter().collect::<Vec<_>>();
+    //     assert_eq!(results.len(), 3);
+    //     assert_eq!(results[0].id, 1);
+    //     assert_eq!(results[1].id, 2);
+    //     assert_eq!(results[2].id, 3);
+    // }
+
     #[test]
-    fn test_any_filter_map() {
+    fn test_any_row_filter_map() {
         let db = EmployeeDB::from_rows(vec![
             Employee {
                 id: 1,
@@ -885,13 +959,13 @@ mod tests_v0_2_1 {
                 is_manager: false,
                 is_admin: false,
                 is_active: true,
-                department: Department::HR,
+                department: Department::Any,
             },
         ])
         .unwrap();
 
         let mut query = db.query();
-        query.department(Department::Any);
+        query.department(Department::Engineering);
         let results = query.execute().unwrap().iter().collect::<Vec<_>>();
         assert_eq!(results.len(), 2);
         assert_eq!(results[0].id, 1);
@@ -899,7 +973,7 @@ mod tests_v0_2_1 {
     }
 
     #[test]
-    fn test_any_option_filter_map() {
+    fn test_any_row_optional_filter_map() {
         let db = WorkerDB::from_rows(vec![
             Worker {
                 id: 1,
@@ -910,29 +984,137 @@ mod tests_v0_2_1 {
             Worker {
                 id: 2,
                 is_admin: false,
+                is_active: None,
+                department: None,
+            },
+            Worker {
+                id: 3,
+                is_admin: false,
+                is_active: Some(true),
+                department: Some(Department::Any),
+            },
+            Worker {
+                id: 4,
+                is_admin: false,
                 is_active: Some(true),
                 department: Some(Department::HR),
             },
+        ])
+        .unwrap();
+
+        let mut query = db.query();
+        query.department(Department::Engineering);
+        let results = query.execute().unwrap().iter().collect::<Vec<_>>();
+        assert_eq!(results.len(), 2);
+        assert_eq!(results[0].id, 1);
+        assert_eq!(results[1].id, 3);
+    }
+}
+
+#[cfg(test)]
+mod tests_v0_3_0 {
+    use super::*;
+
+    #[derive(Debug, VennDB)]
+    pub struct Worker {
+        #[venndb(key)]
+        id: u32,
+        is_admin: bool,
+        is_active: Option<bool>,
+        #[venndb(filter, any)]
+        department: Option<Department>,
+    }
+
+    // regression test: <https://github.com/plabayo/venndb/issues/5>
+    #[test]
+    fn test_any_row_optional_filter_map_white_rabbit() {
+        let db = WorkerDB::from_rows(vec![
+            Worker {
+                id: 1,
+                is_admin: false,
+                is_active: Some(true),
+                department: Some(Department::Engineering),
+            },
             Worker {
-                id: 3,
+                id: 2,
                 is_admin: false,
                 is_active: None,
                 department: None,
             },
+            Worker {
+                id: 3,
+                is_admin: false,
+                is_active: Some(true),
+                department: Some(Department::Any),
+            },
+            Worker {
+                id: 4,
+                is_admin: false,
+                is_active: Some(true),
+                department: Some(Department::HR),
+            },
         ])
         .unwrap();
 
         let mut query = db.query();
-        query.department(Department::Any);
+        query.department(Department::Marketing);
         let results = query.execute().unwrap().iter().collect::<Vec<_>>();
-        assert_eq!(results.len(), 3);
-        assert_eq!(results[0].id, 1);
-        assert_eq!(results[1].id, 2);
-        assert_eq!(results[2].id, 3);
+        assert_eq!(results.len(), 1);
+        assert_eq!(results[0].id, 3);
+    }
+}
+
+#[cfg(test)]
+mod tests_v0_4 {
+    use super::*;
+
+    #[derive(Debug, VennDB)]
+    #[venndb(validator = worker_validator)]
+    pub struct Worker {
+        #[venndb(key)]
+        id: u32,
+        is_admin: bool,
+        is_active: Option<bool>,
+        #[venndb(filter, any)]
+        department: Option<Department>,
+    }
+
+    fn worker_validator(worker: &Worker) -> bool {
+        worker.id > 0 && (worker.is_active.unwrap_or_default() || !worker.is_admin)
     }
 
     #[test]
-    fn test_any_row_filter_map() {
+    fn test_any_filter_map() {
+        let db = EmployeeDB::from_rows(vec![
+            Employee {
+                id: 1,
+                name: "Alice".to_string(),
+                is_manager: true,
+                is_admin: false,
+                is_active: true,
+                department: Department::Engineering,
+            },
+            Employee {
+                id: 2,
+                name: "Bob".to_string(),
+                is_manager: false,
+                is_admin: false,
+                is_active: true,
+                department: Department::HR,
+            },
+        ])
+        .unwrap();
+
+        let mut query = db.query();
+        query.department(Department::Any);
+
+        // no row matches the filter,
+        // given all rows have an explicit department value
+        assert!(query.execute().is_none());
+    }
+
+    #[test]
+    fn test_any_filter_map_match() {
         let db = EmployeeDB::from_rows(vec![
             Employee {
                 id: 1,
@@ -954,15 +1136,14 @@ mod tests_v0_2_1 {
         .unwrap();
 
         let mut query = db.query();
-        query.department(Department::Engineering);
-        let results = query.execute().unwrap().iter().collect::<Vec<_>>();
-        assert_eq!(results.len(), 2);
-        assert_eq!(results[0].id, 1);
-        assert_eq!(results[1].id, 2);
+        query.department(Department::Any);
+
+        let employee = query.execute().unwrap().any();
+        assert_eq!(employee.id, 2);
     }
 
     #[test]
-    fn test_any_row_optional_filter_map() {
+    fn test_any_option_filter_map() {
         let db = WorkerDB::from_rows(vec![
             Worker {
                 id: 1,
@@ -973,51 +1154,97 @@ mod tests_v0_2_1 {
             Worker {
                 id: 2,
                 is_admin: false,
+                is_active: Some(true),
+                department: Some(Department::HR),
+            },
+            Worker {
+                id: 3,
+                is_admin: false,
                 is_active: None,
                 department: None,
             },
+        ])
+        .unwrap();
+
+        let mut query = db.query();
+        query.department(Department::Any);
+
+        // no row matches the filter,
+        // given all rows have an explicit department value
+        assert!(query.execute().is_none());
+    }
+
+    #[test]
+    fn test_any_option_filter_map_match() {
+        let db = WorkerDB::from_rows(vec![
             Worker {
-                id: 3,
+                id: 1,
                 is_admin: false,
                 is_active: Some(true),
-                department: Some(Department::Any),
+                department: Some(Department::Engineering),
             },
             Worker {
-                id: 4,
+                id: 2,
                 is_admin: false,
                 is_active: Some(true),
-                department: Some(Department::HR),
+                department: Some(Department::Any),
+            },
+            Worker {
+                id: 3,
+                is_admin: false,
+                is_active: None,
+                department: None,
             },
         ])
         .unwrap();
 
         let mut query = db.query();
-        query.department(Department::Engineering);
-        let results = query.execute().unwrap().iter().collect::<Vec<_>>();
-        assert_eq!(results.len(), 2);
-        assert_eq!(results[0].id, 1);
-        assert_eq!(results[1].id, 3);
+        query.department(Department::Any);
+
+        let employee = query.execute().unwrap().any();
+        assert_eq!(employee.id, 2);
     }
-}
 
-#[cfg(test)]
-mod tests_v0_2_2 {
-    use super::*;
+    #[test]
+    fn test_worker_db_valid_rows_append() {
+        let mut db = WorkerDB::default();
 
-    #[derive(Debug, VennDB)]
-    pub struct Worker {
-        #[venndb(key)]
-        id: u32,
-        is_admin: bool,
-        is_active: Option<bool>,
-        #[venndb(filter, any)]
-        department: Option<Department>,
+        db.append(Worker {
+            id: 1,
+            is_admin: false,
+            is_active: Some(true),
+            department: Some(Department::Engineering),
+        })
+        .unwrap();
+
+        db.append(Worker {
+            id: 2,
+            is_admin: false,
+            is_active: None,
+            department: None,
+        })
+        .unwrap();
+
+        db.append(Worker {
+            id: 3,
+            is_admin: false,
+            is_active: Some(true),
+            department: Some(Department::Any),
+        })
+        .unwrap();
+
+        db.append(Worker {
+            id: 4,
+            is_admin: true,
+            is_active: Some(true),
+            department: Some(Department::HR),
+        })
+        .unwrap();
     }
 
-    // regression test: <https://github.com/plabayo/venndb/issues/5>
     #[test]
-    fn test_any_row_optional_filter_map_white_rabbit() {
-        let db = WorkerDB::from_rows(vec![
+    fn test_worker_db_valid_rows_from_iter() {
+        WorkerDB::from_iter([
             Worker {
                 id: 1,
                 is_admin: false,
@@ -1038,17 +1265,89 @@ mod tests_v0_2_2 {
             },
             Worker {
                 id: 4,
-                is_admin: false,
+                is_admin: true,
                 is_active: Some(true),
                 department: Some(Department::HR),
             },
         ])
         .unwrap();
+    }
 
-        let mut query = db.query();
-        query.department(Department::Marketing);
-        let results = query.execute().unwrap().iter().collect::<Vec<_>>();
-        assert_eq!(results.len(), 1);
-        assert_eq!(results[0].id, 3);
+    #[test]
+    fn test_worker_db_invalid_rows_append() {
+        let mut db = WorkerDB::default();
+
+        assert_eq!(
+            WorkerDBErrorKind::InvalidRow,
+            db.append(Worker {
+                id: 0,
+                is_admin: false,
+                is_active: None,
+                department: Some(Department::Engineering),
+            })
+            .unwrap_err()
+            .kind()
+        );
+
+        assert_eq!(
+            WorkerDBErrorKind::InvalidRow,
+            db.append(Worker {
+                id: 1,
+                is_admin: true,
+                is_active: Some(false),
+                department: Some(Department::Engineering),
+            })
+            .unwrap_err()
+            .kind()
+        );
+
+        assert_eq!(
+            WorkerDBErrorKind::InvalidRow,
+            db.append(Worker {
+                id: 2,
+                is_admin: true,
+                is_active: None,
+                department: Some(Department::Engineering),
+            })
+            .unwrap_err()
+            .kind()
+        );
+    }
+
+    #[test]
+    fn test_worker_db_invalid_rows_from_iter() {
+        assert_eq!(
+            WorkerDBErrorKind::InvalidRow,
+            WorkerDB::from_iter(
+                [Worker {
+                    id: 0,
+                    is_admin: false,
+                    is_active: None,
+                    department: Some(Department::Engineering),
+                },]
+                .into_iter(),
+            )
+            .unwrap_err()
+            .kind()
+        );
+    }
+
+    #[test]
+    fn test_employee_db_append_invalid_row() {
+        let mut db = EmployeeDB::default();
+
+        assert_eq!(
+            EmployeeDBErrorKind::InvalidRow,
+            db.append(Employee {
+                id: 0,
+                name: "Alice".to_string(),
+                is_manager: true,
+                is_admin: false,
+                is_active: true,
+                department: Department::Engineering,
+            })
+            .unwrap_err()
+            .kind()
+        );
     }
 }


Test patch:
diff --git a/venndb-usage/tests/compiles/derive_struct_all_the_things.rs b/venndb-usage/tests/compiles/derive_struct_all_the_things.rs
index f98b6a5..e207d04 100644
--- a/venndb-usage/tests/compiles/derive_struct_all_the_things.rs
+++ b/venndb-usage/tests/compiles/derive_struct_all_the_things.rs
@@ -1,7 +1,7 @@
 use venndb::{Any, VennDB};
 
 #[derive(Debug, VennDB)]
-#[venndb(name = "EmployeeSheet")]
+#[venndb(name = "EmployeeSheet", validator = employee_validator)]
 struct Employee {
     #[venndb(key)]
     id: u32,
@@ -27,6 +27,10 @@ pub enum Department {
     HR,
 }
 
+fn employee_validator(employee: &Employee) -> bool {
+    employee.id > 0 && !employee.name.is_empty()
+}
+
 impl Any for Department {
     fn is_any(&self) -> bool {
         self == &Department::Any
diff --git a/venndb-usage/tests/compiles/derive_struct_with_validator.rs b/venndb-usage/tests/compiles/derive_struct_with_validator.rs
new file mode 100644
index 0000000..1c41620
--- /dev/null
+++ b/venndb-usage/tests/compiles/derive_struct_with_validator.rs
@@ -0,0 +1,32 @@
+use venndb::VennDB;
+
+#[derive(Debug, VennDB)]
+#[venndb(validator = sealed::employee_validator)]
+struct Employee {
+    pub id: u32,
+    pub name: String,
+    pub is_manager: bool,
+    pub is_admin: bool,
+    pub is_active: bool,
+    pub department: Department,
+}
+
+#[derive(Debug)]
+pub enum Department {
+    Engineering,
+    Sales,
+    Marketing,
+    HR,
+}
+
+mod sealed {
+    use super::Employee;
+
+    pub(super) fn employee_validator(employee: &Employee) -> bool {
+        employee.id > 0 && !employee.name.is_empty()
+    }
+}
+
+fn main() {
+    let _ = EmployeeDB::new();
+}
diff --git a/venndb-usage/tests/fails/derive_struct_with_validator_str.rs b/venndb-usage/tests/fails/derive_struct_with_validator_str.rs
new file mode 100644
index 0000000..21f378f
--- /dev/null
+++ b/venndb-usage/tests/fails/derive_struct_with_validator_str.rs
@@ -0,0 +1,28 @@
+use venndb::VennDB;
+
+#[derive(Debug, VennDB)]
+#[venndb(validator = "employee_validator")]
+struct Employee {
+    pub id: u32,
+    pub name: String,
+    pub is_manager: bool,
+    pub is_admin: bool,
+    pub is_active: bool,
+    pub department: Department,
+}
+
+#[derive(Debug)]
+pub enum Department {
+    Engineering,
+    Sales,
+    Marketing,
+    HR,
+}
+
+fn employee_validator(employee: &Employee) -> bool {
+    employee.id > 0 && !employee.name.is_empty()
+}
+
+fn main() {
+    let _ = EmployeeDB::new();
+}
diff --git a/venndb-usage/tests/fails/derive_struct_with_validator_str.stderr b/venndb-usage/tests/fails/derive_struct_with_validator_str.stderr
new file mode 100644
index 0000000..055ce8a
--- /dev/null
+++ b/venndb-usage/tests/fails/derive_struct_with_validator_str.stderr
@@ -0,0 +1,5 @@
+error: Expected path attribute, found "employee_validator" attribute (literal)
+ --> tests/fails/derive_struct_with_validator_str.rs:4:22
+  |
+4 | #[venndb(validator = "employee_validator")]
+  |                      ^^^^^^^^^^^^^^^^^^^^