internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Error while processing mail - CELERY
### Description

I have problem with Dev branch with Celery. When Paperless consume email with attacment, error is occure :

 Error while processing mail 76999: consume_file() got an **unexpected keyword argument 'task_name'**

### Steps to reproduce

1. Create mail rule
2. Error is occure

### Webserver logs

```bash
[2022-11-06 00:03:18,951] [DEBUG] [paperless_mail] Rule ***@gmail.com.***: Processing mail *** from ***@mail.com with 1 attachment(s)

[2022-11-06 00:03:18,954] [INFO] [paperless_mail] Rule ***@gmail.com.***: Consuming attachment ***.pdf from mail *** from ***@mail.com

[2022-11-06 00:03:18,955] [ERROR] [paperless_mail] Rule ***@gmail.com.***: Error while processing mail 76999: consume_file() got an unexpected keyword argument 'task_name'

Traceback (most recent call last):

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 318, in handle_mail_rule

    processed_files = self.handle_message(message, rule)

  File "/usr/src/paperless/src/paperless_mail/mail.py", line 417, in handle_message

    consume_file.delay(

  File "/usr/local/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay

    return self.apply_async(args, kwargs)

  File "/usr/local/lib/python3.9/site-packages/celery/app/task.py", line 540, in apply_async

    check_arguments(*(args or ()), **(kwargs or {}))

TypeError: consume_file() got an unexpected keyword argument 'task_name'
```


### Browser logs

_No response_

### Paperless-ngx version

dev

### Host OS

Ubuntu

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Thanks :) looks like just a dangling issue from the migration to celery

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -425,7 +425,6 @@ def handle_message(self, message, rule) -> int:
                     else None,
                     override_document_type_id=doc_type.id if doc_type else None,
                     override_tag_ids=tag_ids,
-                    task_name=att.filename[:100],
                 )
 
                 processed_attachments += 1


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Filtering is inconsistent: tag exclusion does not apply with search term
### Description

Filtering documents is not working as expected in some cases.

When filtering for tags, it is possible to include/exclude or exactly match tags. That functionality is lost when using a search term at the same time. I assume that this is not intended behaviour.

### Steps to reproduce

1. Go to `Documents`
2. Enter a search term for filtering documents by that search term. This works fine.
3. Exclude a tag from the search results. The tag is not excluded, when a search term was entered.

### Webserver logs

```bash
Does not apply
```


### Browser logs

_No response_

### Paperless-ngx version

1.10.2

### Host OS

Ubuntu 22.04

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I noticed that this only occurs in the extended search, which does include serching for tags (it seems). Maybe the searching behaviour could be added to the documentation. I can create a PR for that.
Interesting and a bit surprising. The frontend just hits the API here, of course and Im surprised that exlclusion (`tags__id__none`) doesnt work even though inclusion (`tags__id__all`) does, need to see if we can track down why

Resolving patch:
diff --git a/src/documents/index.py b/src/documents/index.py
--- a/src/documents/index.py
+++ b/src/documents/index.py
@@ -143,6 +143,9 @@ def _get_query_filter(self):
             elif k == "tags__id__all":
                 for tag_id in v.split(","):
                     criterias.append(query.Term("tag_id", tag_id))
+            elif k == "tags__id__none":
+                for tag_id in v.split(","):
+                    criterias.append(query.Not(query.Term("tag_id", tag_id)))
             elif k == "document_type__id":
                 criterias.append(query.Term("type_id", v))
             elif k == "correspondent__isnull":


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[Bug] cannot save Mail Rule with "mail and attachment as seperate documents" in 1.11.1
      Maybe it's just me, but I cannot save Mail Rule with "mail and attachment as seperate documents".

_Originally posted by @Limerick-gh in https://github.com/paperless-ngx/paperless-ngx/discussions/2265#discussioncomment-4557234_
    
[Bug] Missing consumption scope options in frontend
### Discussed in https://github.com/paperless-ngx/paperless-ngx/discussions/2265

<div type='discussions-op-text'>

<sup>Originally posted by **morremeyer** December 30, 2022</sup>
With #2000, frontend configuration for mail consumption was added.
With #848, at about the same time, email body & .eml file consumption was added.

#848 added the **consumption scope** for email consumption (see https://github.com/p-h-a-i-l/paperless-ngx/blob/0fda35723d62275a5beb783cbf9061d4d4a15703/src/paperless_mail/models.py#L59-L65) to decide between consuming:

* only the attachments
* the full email as .eml
* the full email as .eml **and** the attachments

The **consumption scope** is not yet configurable on the frontend. I'd be really happy if it were configurable in the frontend in a future version.

I'm pretty sure someone already has that planned, but I couldn't find an issue or discussion for it, so I'm opening this one to track this request.</div>


Discussion:

Closed in branch by https://github.com/paperless-ngx/paperless-ngx/commit/06dab340dde1533dd3c536002f1ff379c61821db

Resolving patch:
diff --git a/src/paperless_mail/serialisers.py b/src/paperless_mail/serialisers.py
--- a/src/paperless_mail/serialisers.py
+++ b/src/paperless_mail/serialisers.py
@@ -86,6 +86,7 @@ class Meta:
             "assign_document_type",
             "order",
             "attachment_type",
+            "consumption_scope",
         ]
 
     def update(self, instance, validated_data):


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Sorting tags by color does not work
### Description

When click on the header field for the "Color" column the order of the tags never changes.

### Steps to reproduce

1. Go to "Tag Management"
2. Click on "Color" table header
3. Little arrow besides header name indicates sorting but no changes in sort order occur


### Webserver logs

```bash
Does not apply. (I think?)
```


### Browser logs

_No response_

### Paperless-ngx version

vaultwarden/server:1.26.0-alpine

### Host OS

OpenMediaVault 6.x (Debian)

### Installation method

Docker - official image

### Browser

Browser Firefox 106.0.3 (64-Bit)

### Configuration changes

_No response_

### Other

I have pretty much a standard installation, afaik. Sorting other columns like tag name does work and the little arrow appears and changes direction after repeated clicks, so my guess would be the data type for the column is wrong or undefined?


Discussion:

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -176,7 +176,7 @@ def get_serializer_class(self):
     permission_classes = (IsAuthenticated,)
     filter_backends = (DjangoFilterBackend, OrderingFilter)
     filterset_class = TagFilterSet
-    ordering_fields = ("name", "matching_algorithm", "match", "document_count")
+    ordering_fields = ("color", "name", "matching_algorithm", "match", "document_count")
 
 
 class DocumentTypeViewSet(ModelViewSet):


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] unable to change mail rule order in new frontend
### Description

i was setting up mail rules and wasn't able to get them to work properly until i realized that in the django admin i was able to set a order for those rules. in the new frontend the order was already displayed how i actually wanted them to be but in the django admin i was able to see they were in the wrong order

### Steps to reproduce

1. go to django admin
2. change mail order rule
3. go to new mail rule frontend
4. mail rule order doesn't match the django admin order
5. (also there is no way to change the order in the new frontend)

### Webserver logs

```bash
n/a
```


### Browser logs

_No response_

### Paperless-ngx version

1.11.4

### Host OS

Debian 11

### Installation method

Bare metal

### Browser

Firefox 108

### Configuration changes

_No response_

### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/paperless_mail/views.py b/src/paperless_mail/views.py
--- a/src/paperless_mail/views.py
+++ b/src/paperless_mail/views.py
@@ -27,7 +27,7 @@ class MailAccountViewSet(ModelViewSet):
 class MailRuleViewSet(ModelViewSet):
     model = MailRule
 
-    queryset = MailRule.objects.all().order_by("pk")
+    queryset = MailRule.objects.all().order_by("order")
     serializer_class = MailRuleSerializer
     pagination_class = StandardPagination
     permission_classes = (IsAuthenticated,)


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Date range view no longer works
### Description

I recently installed the latest release 1.12.0. Since this upgrade, the view for a date period no longer works correctly. An error message always appears. The error occurs with my saved view as well as when manually selecting the filter for a date period. The other filters, such as for a document type, work. With the previous version 1.11.3 everything still worked.

![Bildschirmfoto 2023-01-25 um 16 43 22](https://user-images.githubusercontent.com/70706251/214608485-3bbfee34-5007-47ce-a284-41bf2014933d.png)

**Browser Request:**
![Bildschirmfoto 2023-01-25 um 16 53 48](https://user-images.githubusercontent.com/70706251/214611137-e9408321-44ac-495a-b273-c874eebd496e.png)

![Bildschirmfoto 2023-01-25 um 16 53 39](https://user-images.githubusercontent.com/70706251/214611177-2b625e8c-234d-4ae7-b95d-35609fa7aab3.png)





### Steps to reproduce

1. Go to Documents
2. select filter "selected on" or "created on"
3. A period is selected

### Webserver logs

```bash
I cannot see any error in the logs
```


### Browser logs

```bash
refer to screenshot in the description
```


### Paperless-ngx version

1.12.0

### Host OS

Raspberry Pi 'arm64'

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

docker-compose.yml

### Other

_No response_


Discussion:
This is related to https://github.com/paperless-ngx/paperless-ngx/pull/2351 , already working on a fix

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -468,14 +468,16 @@ def comments(self, request, pk=None):
 class SearchResultSerializer(DocumentSerializer):
     def to_representation(self, instance):
         doc = Document.objects.get(id=instance["id"])
-        commentTerm = instance.results.q.subqueries[0]
-        comments = ",".join(
-            [
-                str(c.comment)
-                for c in Comment.objects.filter(document=instance["id"])
-                if commentTerm.text in c.comment
-            ],
-        )
+        comments = ""
+        if hasattr(instance.results.q, "subqueries"):
+            commentTerm = instance.results.q.subqueries[0]
+            comments = ",".join(
+                [
+                    str(c.comment)
+                    for c in Comment.objects.filter(document=instance["id"])
+                    if commentTerm.text in c.comment
+                ],
+            )
         r = super().to_representation(doc)
         r["__search_hit__"] = {
             "score": instance.score,


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Extended search in comments doesn’t work
### Description

The search in the comments of the documents introduced with version 1.12.0 does not work for me.

### Steps to reproduce

Save any word under comments for any document and then search for it.

### Webserver logs

```bash
No entries
```


### Browser logs

```bash
No entries
```


### Paperless-ngx version

1.12.0

### Host OS

Synology DSM

### Installation method

Docker - official image

### Browser

Safari, Chrome

### Configuration changes

None

### Other

_No response_


Discussion:
Can you [rebuild the index](https://docs.paperless-ngx.com/administration/#index)?  Then make a comment and check
Ah, perhaps we need to increment the index version to [trigger a reindex](https://github.com/paperless-ngx/paperless-ngx/blob/a4535c11e46efdf638ad0800ab5f1fba237a2623/docker/docker-prepare.sh#L86)?
I have rebuilt the index. After that, it finds all the comments entered so far. But when you enter a new comment, it is not found - presumably only after a new indexing. 

And I found this warning in the log, which I can't relate to: 

`[2023-01-25 20:54:14,052] [WARNING] [paperless.parsing.tesseract] Error while reading metadata {http://www.aiim.org/pdfa/ns/extension/}schemas: {'\n\t\t\t\t\t\t'}. Error: 'http://www.aiim.org/pdfa/ns/extension/'`

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -445,6 +445,10 @@ def comments(self, request, pk=None):
                 )
                 c.save()
 
+                from documents import index
+
+                index.add_or_update_document(self.get_object())
+
                 return Response(self.getComments(doc))
             except Exception as e:
                 logger.warning(f"An error occurred saving comment: {str(e)}")
@@ -456,6 +460,11 @@ def comments(self, request, pk=None):
         elif request.method == "DELETE":
             comment = Comment.objects.get(id=int(request.GET.get("id")))
             comment.delete()
+
+            from documents import index
+
+            index.add_or_update_document(self.get_object())
+
             return Response(self.getComments(doc))
 
         return Response(


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Multi-word comment search strings not working in 1.12.1
### Description

if 1 word is searched in advanced search "example" paperless will find the word in commits

if 2 words are searched in commits "example two" this ends in error:
Fehler beim Laden des Dokuments: 'Or' object has no attribute 'text'
![capture_001_27012023_135402](https://user-images.githubusercontent.com/123741690/215092032-520a101d-cf42-4a98-b6bb-0f04b5ba2af7.jpg)
![capture_001_27012023_140154](https://user-images.githubusercontent.com/123741690/215093269-14da0143-e534-4fc8-b7b0-b98070ce16d9.jpg)


### Steps to reproduce

Go to expanded search
search 2 words
error occured

### Webserver logs

```bash
no related logs
```


### Browser logs

_No response_

### Paperless-ngx version

1.12.1

### Host OS

Docker official Synologie NAS

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Ah thanks, I will track this one down

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -477,21 +477,14 @@ def comments(self, request, pk=None):
 class SearchResultSerializer(DocumentSerializer):
     def to_representation(self, instance):
         doc = Document.objects.get(id=instance["id"])
-        comments = ""
-        if hasattr(instance.results.q, "subqueries"):
-            commentTerm = instance.results.q.subqueries[0]
-            comments = ",".join(
-                [
-                    str(c.comment)
-                    for c in Comment.objects.filter(document=instance["id"])
-                    if commentTerm.text in c.comment
-                ],
-            )
+        comments = ",".join(
+            [str(c.comment) for c in Comment.objects.filter(document=instance["id"])],
+        )
         r = super().to_representation(doc)
         r["__search_hit__"] = {
             "score": instance.score,
             "highlights": instance.highlights("content", text=doc.content),
-            "comment_highlights": instance.highlights("content", text=comments)
+            "comment_highlights": instance.highlights("comments", text=comments)
             if doc
             else None,
             "rank": instance.rank,


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] document_exporter: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/paperless/paperless-exportmwjrtave'
### Description

when using the document_exporter with "-z" flag for creating zip archives the document exporter assumes the directory /tmp/paperless to be already existing.

document_exporter.py
```python
        # If zipping, save the original target for later and
        # get a temporary directory for the target
        temp_dir = None
        original_target = None
        if zip_export:
            original_target = self.target
            temp_dir = tempfile.TemporaryDirectory(
                dir=settings.SCRATCH_DIR,
                prefix="paperless-export",
            )
            self.target = Path(temp_dir.name).resolve()

```

I think it would be a good idea to check the existance of this directory beforehand like it is done in 
consumer.py

```python
    def pre_check_directories(self):
        os.makedirs(settings.SCRATCH_DIR, exist_ok=True)
```

KR

### Steps to reproduce

Call the document exporter:

`paperless_ngx__directory/.venv/bin/python3 paperless_ngx__directory/src/manage.py document_exporter /mnt/backuptarget -c -d -z`

### Webserver logs

```bash
no logs needed
```


### Browser logs

_No response_

### Paperless-ngx version

1.13.0

### Host OS

Ubuntu 22.04

### Installation method

Bare metal

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
I haven't tested it, but it's probably good and quick to do.

Resolving patch:
diff --git a/src/documents/management/commands/document_exporter.py b/src/documents/management/commands/document_exporter.py
--- a/src/documents/management/commands/document_exporter.py
+++ b/src/documents/management/commands/document_exporter.py
@@ -156,11 +156,13 @@ def handle(self, *args, **options):
         zip_export: bool = options["zip"]
 
         # If zipping, save the original target for later and
-        # get a temporary directory for the target
+        # get a temporary directory for the target instead
         temp_dir = None
         original_target = None
         if zip_export:
             original_target = self.target
+
+            os.makedirs(settings.SCRATCH_DIR, exist_ok=True)
             temp_dir = tempfile.TemporaryDirectory(
                 dir=settings.SCRATCH_DIR,
                 prefix="paperless-export",


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Display of View and Edit permissions problem
Issue:

I have a group, and two users in that group. Assigning just edit permissions to the group on a document, and then reopening results in the following:



![Image](https://user-images.githubusercontent.com/17569239/222853790-a77dea65-c74d-49a8-a635-2bf52b54de66.png)

This is when logged in as "superuser".

* the tag of "superuser" seems messed up.
* Just entering work as the group in edit populates all this data. It is technically correct, when I give edit permissions to a group, then these are the implied permissions (edit for the users of the group, and view permissions for the group and its users). However I wouldn't expect them to show up here, since I did not explicitly configure anything expect editing for the group "work".


Discussion:

Resolving patch:
diff --git a/src/documents/permissions.py b/src/documents/permissions.py
--- a/src/documents/permissions.py
+++ b/src/documents/permissions.py
@@ -63,6 +63,7 @@ def set_permissions_for_object(permissions, object):
         users_to_remove = get_users_with_perms(
             object,
             only_with_perms_in=[permission],
+            with_group_users=False,
         )
         if len(users_to_add) > 0 and len(users_to_remove) > 0:
             users_to_remove = users_to_remove.difference(users_to_add)
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -161,6 +161,7 @@ def get_permissions(self, obj):
                 "users": get_users_with_perms(
                     obj,
                     only_with_perms_in=[view_codename],
+                    with_group_users=False,
                 ).values_list("id", flat=True),
                 "groups": get_groups_with_only_permission(
                     obj,
@@ -171,6 +172,7 @@ def get_permissions(self, obj):
                 "users": get_users_with_perms(
                     obj,
                     only_with_perms_in=[change_codename],
+                    with_group_users=False,
                 ).values_list("id", flat=True),
                 "groups": get_groups_with_only_permission(
                     obj,
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -964,6 +964,7 @@ def get(self, request, format=None):
                     "id": user.id,
                     "username": user.username,
                     "is_superuser": user.is_superuser,
+                    "groups": user.groups.values_list("id", flat=True),
                 },
                 "settings": ui_settings,
                 "permissions": roles,


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Unable to edit permissions as super user on documents owned by other users
Issue:

1. Create a super user
2. Create another user
3. Add docs while logged in as the super user (owner=super user)
4. Using bulk edit (or i assume just editing details works too), set owner to the other user.

Observed Behavior:

Currently logged in super user is no longer able to edit the document (and reset owner for example)

Expected behavior:

Super user should be able to edit all permissions (?)

Notes:

This is what I'd expect. My closest comparison is the super user on unix, who is also allowed to do almost anything, pretty much.


Discussion:
Yea this is intentional at the moment, we can change it if we think superusers should be able to do this, I guess your vote would be yes, Im OK with that

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -931,8 +931,11 @@ def get(self, request, format=None):
         roles = map(lambda perm: re.sub(r"^\w+.", "", perm), user.get_all_permissions())
         return Response(
             {
-                "user_id": user.id,
-                "username": user.username,
+                "user": {
+                    "id": user.id,
+                    "username": user.username,
+                    "is_superuser": user.is_superuser,
+                },
                 "settings": ui_settings,
                 "permissions": roles,
             },


Test patch: