Repository: Mailu/Mailu

Issue:
config-import  [KeyError] 'id'
Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).

To be able to help you best, we need some more information.

## Before you open your issue
- [ ] Check if no issue or pull-request for this already exists.
- [ ] Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
- [ ] You understand `Mailu` is made by volunteers in their **free time** — be concise, civil and accept that delays can occur.
- [ ] The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.

## Environment & Versions
### Environment
 - [X] docker-compose 
 - [ ] kubernetes
 - [ ] docker swarm

### Versions
To find your version, get the image name of a mailu container and read  the version from the tag (example for version 1.7).
```
$> docker ps -a | grep mailu
140b09d4b09c    mailu/roundcube:1.7    "docker-php-entrypoi…"    2 weeks ago    Up 2 days (healthy)    80/tcp
$> grep MAILU_VERSION docker-compose.yml mailu.env
```
version 1.9

## Description
Further explain the bug in a few words. It should be clear what the unexpected behaviour is.  Share it in an easy-to-understand language.

Reciving Error: [KeyError] 'id' , when trying to import configuration, with fetches.

## Replication Steps
Steps for replicating your issue

docker exec -i $(docker-compose ps -q admin) flask mailu config-import -uv < import-user.yml

domain:
  - name: example.com

 user:
  - email: testimport@example.com
    displayed_name: 'Test'
    enable_imap: true
    enable_pop: false
    enabled: true
    fetches:
      - comment: ''
        error: null
        host: 0.0.0.0
        keep: true
        last_check: '2022-10-25T14:26:20.876898Z'
        password: 'test'
        port: 993
        protocol: imap
        tls: true
        username: admin@example.com
    password: 'test'
    hash_password: false
    quota_bytes: 10000000
    spam_enabled: true
    spam_mark_as_read: false


## Expected behaviour
Explain what results you expected - be as specific as possible. Just saying "it doesn’t work as expected" is not useful. It's also helpful to describe what you actually experienced.

The expected behaviours is to be able to create a new user, while fetching other mailbox.

## Logs
Often it is very useful to include log fragments of the involved component. You can get the logs via `docker logs <container name> --tail 1000`. For example for the admin container:
`docker logs mailu_admin_1 --tail 1000`
or using docker-compose `docker-compose -f /mailu/docker-compose.yml logs --tail 1000 admin`
If you can find the relevant section, please share only the parts that seem relevant. If you have any logs, please enclose them in code tags, like so:
````markdown
```
Your logs here!
```
````



Discussion:
You can just add an id to work around this: `id: 1`
I'll see if it is possible to create a new id if it's missing.
The id is needed to know if you want to add or replace a new item.
Thanks for the reply, when I add id: 1.
I'm receving the following error:
Error: [AttributeError] 'User' object has no attribute 'id'
You need to add the id to the fetch item, not the user.
Please just create a user with a fetch using the web interface and export your configuration using "config-export" to see how it should look like.
Hi,

I have exported the configuration, and still receiving the same error:
Error: [AttributeError] 'User' object has no attribute 'id'

In the example you can see that the id is already in the fetch item:

--- 
domain: 
  - 
    name: example.com
user: 
  - 
    displayed_name: Test
    email: newtest@example.com
    enable_imap: true
    enable_pop: false
    enabled: true
    fetches: 
      - 
        comment: ""
        error: ~
        host: "example"
        id: 3
        keep: true
        last_check: ~
        password: example
        port: 993
        protocol: imap
        tls: false
        username: example@example.com
    forward_destination: []
    forward_enabled: false
    forward_keep: true
    global_admin: false
    manager_of: []
    password: "example"
    quota_bytes: 1000000000
    reply_body: ~
    reply_enabled: false
    reply_enddate: "2999-12-31"
    reply_startdate: "1900-01-01"
    reply_subject: ~
    spam_enabled: true
    spam_mark_as_read: true
    spam_threshold: 80

I fear the bugfix has opened another bug. I'll try to fix this asap.

Resolving patch:
diff --git a/core/admin/mailu/schemas.py b/core/admin/mailu/schemas.py
--- a/core/admin/mailu/schemas.py
+++ b/core/admin/mailu/schemas.py
@@ -5,6 +5,7 @@
 from collections import Counter
 from datetime import timezone
 
+import inspect
 import json
 import logging
 import yaml
@@ -669,20 +670,15 @@ class Storage:
 
     context = {}
 
-    def _bind(self, key, bind):
-        if bind is True:
-            return (self.__class__, key)
-        if isinstance(bind, str):
-            return (get_schema(self.recall(bind).__class__), key)
-        return (bind, key)
-
-    def store(self, key, value, bind=None):
+    def store(self, key, value):
         """ store value under key """
-        self.context.setdefault('_track', {})[self._bind(key, bind)]= value
+        key = f'{self.__class__.__name__}.{key}'
+        self.context.setdefault('_track', {})[key] = value
 
-    def recall(self, key, bind=None):
+    def recall(self, key):
         """ recall value from key """
-        return self.context['_track'][self._bind(key, bind)]
+        key = f'{self.__class__.__name__}.{key}'
+        return self.context['_track'][key]
 
 class BaseOpts(SQLAlchemyAutoSchemaOpts):
     """ Option class with sqla session
@@ -790,10 +786,16 @@ def hide(self, data):
             for key, value in data.items()
         }
 
-    def _call_and_store(self, *args, **kwargs):
-        """ track current parent field for pruning """
-        self.store('field', kwargs['field_name'], True)
-        return super()._call_and_store(*args, **kwargs)
+    def get_parent(self):
+        """ helper to determine parent of current object """
+        for x in inspect.stack():
+            loc = x[0].f_locals
+            if 'ret_d' in loc:
+                if isinstance(loc['self'], MailuSchema):
+                    return self.context.get('config'), loc['attr_name']
+                else:
+                    return loc['self'].get_instance(loc['ret_d']), loc['attr_name']
+        return None, None
 
     # this is only needed to work around the declared attr "email" primary key in model
     def get_instance(self, data):
@@ -803,9 +805,13 @@ def get_instance(self, data):
         if keys := getattr(self.Meta, 'primary_keys', None):
             filters = {key: data.get(key) for key in keys}
             if None not in filters.values():
-                res= self.session.query(self.opts.model).filter_by(**filters).first()
-                return res
-        res= super().get_instance(data)
+                try:
+                    res = self.session.query(self.opts.model).filter_by(**filters).first()
+                except sqlalchemy.exc.StatementError as exc:
+                    raise ValidationError(f'Invalid {keys[0]}: {data.get(keys[0])!r}', data.get(keys[0])) from exc
+                else:
+                    return res
+        res = super().get_instance(data)
         return res
 
     @pre_load(pass_many=True)
@@ -829,6 +835,10 @@ def _patch_many(self, items, many, **kwargs): # pylint: disable=unused-argument
         want_prune = []
         def patch(count, data):
 
+            # we only process objects here
+            if type(data) is not dict:
+                raise ValidationError(f'Invalid item. {self.Meta.model.__tablename__.title()} needs to be an object.', f'{data!r}')
+
             # don't allow __delete__ coming from input
             if '__delete__' in data:
                 raise ValidationError('Unknown field.', f'{count}.__delete__')
@@ -882,10 +892,10 @@ def set_default(key, value):
         ]
 
         # remember if prune was requested for _prune_items@post_load
-        self.store('prune', bool(want_prune), True)
+        self.store('prune', bool(want_prune))
 
         # remember original items to stabilize password-changes in _add_instance@post_load
-        self.store('original', items, True)
+        self.store('original', items)
 
         return items
 
@@ -909,23 +919,18 @@ def _patch_item(self, data, many, **kwargs): # pylint: disable=unused-argument
         # stabilize import of auto-increment primary keys (not required),
         # by matching import data to existing items and setting primary key
         if not self._primary in data:
-            parent = self.recall('parent')
+            parent, field = self.get_parent()
             if parent is not None:
-                for item in getattr(parent, self.recall('field', 'parent')):
+                for item in getattr(parent, field):
                     existing = self.dump(item, many=False)
                     this = existing.pop(self._primary)
                     if data == existing:
-                        instance = item
+                        self.instance = item
                         data[self._primary] = this
                         break
 
         # try to load instance
         instance = self.instance or self.get_instance(data)
-
-        # remember instance as parent for pruning siblings
-        if not self.Meta.sibling and self.context.get('update'):
-            self.store('parent', instance)
-
         if instance is None:
 
             if '__delete__' in data:
@@ -1001,7 +1006,7 @@ def _prune_items(self, items, many, **kwargs): # pylint: disable=unused-argument
             return items
 
         # get prune flag from _patch_many@pre_load
-        want_prune = self.recall('prune', True)
+        want_prune = self.recall('prune')
 
         # prune: determine if existing items in db need to be added or marked for deletion
         add_items = False
@@ -1018,16 +1023,17 @@ def _prune_items(self, items, many, **kwargs): # pylint: disable=unused-argument
                 del_items = True
 
         if add_items or del_items:
-            parent = self.recall('parent')
+            parent, field = self.get_parent()
             if parent is not None:
                 existing = {item[self._primary] for item in items if self._primary in item}
-                for item in getattr(parent, self.recall('field', 'parent')):
+                for item in getattr(parent, field):
                     key = getattr(item, self._primary)
                     if key not in existing:
                         if add_items:
                             items.append({self._primary: key})
                         else:
-                            items.append({self._primary: key, '__delete__': '?'})
+                            if self.context.get('update'):
+                                self.opts.sqla_session.delete(self.instance or self.get_instance({self._primary: key}))
 
         return items
 
@@ -1048,7 +1054,7 @@ def _add_instance(self, item, many, **kwargs): # pylint: disable=unused-argument
         # did we hash a new plaintext password?
         original = None
         pkey = getattr(item, self._primary)
-        for data in self.recall('original', True):
+        for data in self.recall('original'):
             if 'hash_password' in data and data.get(self._primary) == pkey:
                 original = data['password']
                 break
@@ -1244,12 +1250,6 @@ def __init__(self, *args, **kwargs):
                 if field in fieldlist:
                     fieldlist[field] = fieldlist.pop(field)
 
-    def _call_and_store(self, *args, **kwargs):
-        """ track current parent and field for pruning """
-        self.store('field', kwargs['field_name'], True)
-        self.store('parent', self.context.get('config'))
-        return super()._call_and_store(*args, **kwargs)
-
     @pre_load
     def _clear_config(self, data, many, **kwargs): # pylint: disable=unused-argument
         """ create config object in context if missing


Test patch: