internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Gmail 'tag' rule removes current label on email
### Description

Some of my emails in Gmail are labeled when received and Gmail supports emails having multiple labels. When paperless imports the email attachment the current label is removed and replaced by the 'Paperless' (my action parameter) label.

Is there any way for the 'Paperless' label to be added as an additional label without removing current labels?

### Steps to reproduce

1. In Gmail labels an email
2. Paperless consumed email with attachment
3. Email now has original label removed and action parameter label added

### Webserver logs

```bash
.
```


### Browser logs

_No response_

### Paperless-ngx version

1.13.0

### Host OS

Synology

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Not 100% a bug exactly.  But it does differ from how the normal IMAP flagging would work.

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -165,7 +165,7 @@ def get_criteria(self):
 
     def post_consume(self, M: MailBox, message_uid: str, parameter: str):
         if re.search(r"gmail\.com$|googlemail\.com$", M._host):
-            M.client.uid("STORE", message_uid, "X-GM-LABELS", self.keyword)
+            M.client.uid("STORE", message_uid, "+X-GM-LABELS", self.keyword)
 
         # AppleMail
         elif self.color:


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Can't view/load certain documents in Chrome (,') - ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION
### Description

I found a bug in viewing Documents that include , AND ' in the file path in Chromium based browsers. (Also happens on freshly installed paperless with default settings)

Editing window opens up but doesn't show document preview.
Attempting to download or preview causes Chromium (Brave in my case) to throw ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION.

This might help solving the issue:
https://www.ryadel.com/en/err_response_headers_multiple_content_disposition-error-google-chrome-fix/

### Steps to reproduce

1. Rename Document "Let's meet every monday, tuesday and friday"
2. Save
3. Reopen document or try to preview/download

or

1. Rename Correspondent "Tony's"
2. Rename Document to "Pizza, Pasta and more"
3. Save
4. Reopen document or try to preview/download

### Webserver logs

```bash
--
```


### Browser logs

```bash
--
```


### Paperless-ngx version

1.13.0

### Host OS

Manjaro Linux x86_64

### Installation method

Docker - official image

### Browser

Chromium Version 111.0.5563.64 (Official Build) Arch Linux (64-bit)

### Configuration changes

docker-compose.env: PAPERLESS_FILENAME_FORMAT={created_year}/{correspondent}/{created}-{title}

### Other

_No response_


Discussion:
The proposed fix from the article is already done is both places we directly control the content setting:
https://github.com/paperless-ngx/paperless-ngx/blob/851b23fb09150981b01f1b5fd4b03ef028f88510/src/documents/views.py#L318

https://github.com/paperless-ngx/paperless-ngx/blob/851b23fb09150981b01f1b5fd4b03ef028f88510/src/documents/views.py#L845

I guess replacing commas with something else, like underscores

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -311,7 +311,11 @@ def file_response(self, pk, request, disposition):
         # Firefox is not able to handle unicode characters in filename field
         # RFC 5987 addresses this issue
         # see https://datatracker.ietf.org/doc/html/rfc5987#section-4.2
-        filename_normalized = normalize("NFKD", filename).encode("ascii", "ignore")
+        # Chromium cannot handle commas in the filename
+        filename_normalized = normalize("NFKD", filename.replace(",", "_")).encode(
+            "ascii",
+            "ignore",
+        )
         filename_encoded = quote(filename)
         content_disposition = (
             f"{disposition}; "


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] error when updating correspondent
### Description

When updating (press save button) an existing correspondent/tag/document type/etc, paperless throws an error, and the dialog box stays open.

please open .har file in chrome to see the interaction with server (payload/response).
[paperless-har-file..zip](https://github.com/paperless-ngx/paperless-ngx/files/11050147/paperless-har-file.zip)


### Steps to reproduce

[paperless.steps.zip](https://github.com/paperless-ngx/paperless-ngx/files/11050909/paperless.steps.zip)


### Webserver logs

```bash
*does not show any related logs on the webserver log.*
```


### Browser logs

```bash
{"error":["Object violates owner / name unique constraint"]}
```


### Paperless-ngx version

Paperless-ngx 1.13.0-dev

### Host OS

Ubuntu 20.4

### Installation method

Docker - official image

### Browser

Chrome

### Configuration changes

_No response_

### Other

_No response_


Discussion:
> `{"error":["Object violates owner / name unique constraint"]}`

This seems like a very clear response.  You can't create a correspondent with the same name, owned by the same person.

The error pop up could include the same thing visible in the logs though for a better experience.
Ah, but I am seeing this when just updating, I'll look into it, thanks

Resolving patch:
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -78,10 +78,11 @@ def validate(self, data):
             if hasattr(self, "user")
             else None
         )
+        pk = self.instance.pk if hasattr(self.instance, "pk") else None
         if ("name" in data or "owner" in data) and self.Meta.model.objects.filter(
             name=name,
             owner=owner,
-        ).exists():
+        ).exclude(pk=pk).exists():
             raise serializers.ValidationError(
                 {"error": "Object violates owner / name unique constraint"},
             )


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Email tag rule inverted for non Gmail labels
### Description

Caused by https://github.com/paperless-ngx/paperless-ngx/pull/3240 which removed the X-GM-LABELS when the server doesn't support it, but (seemingly) kept the “NOT“ before it.

### Steps to reproduce

1. Create a rule with tag for non-GM
2. Should fetch untagged
3. Only fetches already tagged, ignoring other mails

Manually adding a tag to a mail caused it to be imported.

### Webserver logs

```bash
Searching folder with criteria (NOT (UNKEYWORD paperless) SINCE xxxxx
```


### Browser logs

_No response_

### Paperless-ngx version

1.14.3

### Host OS

Debian

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/paperless_mail/mail.py b/src/paperless_mail/mail.py
--- a/src/paperless_mail/mail.py
+++ b/src/paperless_mail/mail.py
@@ -168,7 +168,7 @@ def get_criteria(self):
             if self.supports_gmail_labels:
                 return AND(NOT(gmail_label=self.keyword), no_keyword=self.keyword)
             else:
-                return NOT(no_keyword=self.keyword)
+                return {"no_keyword": self.keyword}
         else:  # pragma: nocover
             raise ValueError("This should never happen.")
 


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] upload status messages are visible by all users
### Description

Upload process of a new document by user A is visible by user B in the dashboard and by a notification. 
If user B tries to open such document, he has no access to that and lands on the 404-page. 
Browser back-button from 404-page navigates again to the document, that can't be accessed by user B, so he lands again on the 404-page. The only solution here is to close paperless and open it again in a new tab.

I see here two issues:
1. upload process from user A shouldn't be seen by user B
2. 404-page back-button navigation issue.

### Steps to reproduce

1. create two users: A and B
2. login with user A in one browser
3. login with user B in another browser
4. upload a document with user A
5. see the uploading process on the dashboard of user B
6. user B should try to open the uploaded document

### Webserver logs

```bash
no logs
```


### Browser logs

_No response_

### Paperless-ngx version

1.17.0

### Host OS

Debian x86_64

### Installation method

Docker - official image

### Browser

Chrome, Safari

### Configuration changes

_No response_

### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/documents/consumer.py b/src/documents/consumer.py
--- a/src/documents/consumer.py
+++ b/src/documents/consumer.py
@@ -90,6 +90,7 @@ def _send_progress(
             "status": status,
             "message": message,
             "document_id": document_id,
+            "owner_id": self.override_owner_id if self.override_owner_id else None,
         }
         async_to_sync(self.channel_layer.group_send)(
             "status_updates",
@@ -118,7 +119,7 @@ def __init__(self):
         self.override_document_type_id = None
         self.override_asn = None
         self.task_id = None
-        self.owner_id = None
+        self.override_owner_id = None
 
         self.channel_layer = get_channel_layer()
 
diff --git a/src/paperless/consumers.py b/src/paperless/consumers.py
--- a/src/paperless/consumers.py
+++ b/src/paperless/consumers.py
@@ -10,6 +10,16 @@ class StatusConsumer(WebsocketConsumer):
     def _authenticated(self):
         return "user" in self.scope and self.scope["user"].is_authenticated
 
+    def _is_owner_or_unowned(self, data):
+        return (
+            (
+                self.scope["user"].is_superuser
+                or self.scope["user"].id == data["owner_id"]
+            )
+            if "owner_id" in data and "user" in self.scope
+            else True
+        )
+
     def connect(self):
         if not self._authenticated():
             raise DenyConnection
@@ -30,4 +40,5 @@ def status_update(self, event):
         if not self._authenticated():
             self.close()
         else:
-            self.send(json.dumps(event["data"]))
+            if self._is_owner_or_unowned(event["data"]):
+                self.send(json.dumps(event["data"]))


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[Security] saved_views API returns (hashed) user password in response
### Description

The response of `GET /api/saved_views/` includes the hashed password of the owner of the saved view.

### Steps to reproduce

```
curl -uuser:pass https://host.com/api/saved_views/ | jq .results[].owner.password
```

### Webserver logs

```bash
-
```


### Browser logs

_No response_

### Paperless-ngx version

1.16.5

### Host OS

Debian GNU/Linux 12

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -476,7 +476,6 @@ class SavedViewSerializer(OwnedObjectSerializer):
 
     class Meta:
         model = SavedView
-        depth = 1
         fields = [
             "id",
             "name",


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Invalid variable detected for "doc_pk"
### Description

Currently, it is not possible to use the new variable "doc_pk" in storage paths. An error is encountered "Invalid variable detected.". I would like to use this variable in my file naming options

![image](https://github.com/paperless-ngx/paperless-ngx/assets/823564/de6a4922-8662-47db-869c-8996139e7589)


### Steps to reproduce

1. Go to "Manage / Storage paths"
2. Click on to "Create"
3. Assign a name to the path
4. Assign a path with the variable "doc_pk" (eg. {doc_pk} or {created_year}/{created}-{doc_pk}

### Webserver logs

```bash
[WARNING] [django.request] Bad Request: /api/storage_paths/1/
```


### Browser logs

```bash
{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":400,"statusText":"Bad Request","url":"http://coyotte:8002/api/storage_paths/","ok":false,"name":"HttpErrorResponse","message":"Http failure response for http://coyotte:8002/api/storage_paths/: 400 Bad Request","error":{"path":["Invalid variable detected."]}}
```


### Paperless-ngx version

1.7.4

### Host OS

Ubuntu 23.04

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

_No response_

### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/documents/serialisers.py b/src/documents/serialisers.py
--- a/src/documents/serialisers.py
+++ b/src/documents/serialisers.py
@@ -835,6 +835,7 @@ def validate_path(self, path):
                 tag_list="tag_list",
                 owner_username="someone",
                 original_name="testfile",
+                dock_pk="doc_pk",
             )
 
         except KeyError as err:


Test patch:
diff --git a/src/documents/tests/test_api.py b/src/documents/tests/test_api.py
--- a/src/documents/tests/test_api.py
+++ b/src/documents/tests/test_api.py
@@ -4463,7 +4463,7 @@ def test_api_storage_path_placeholders(self):
                     "/{created_month_name_short}/{created_day}/{added}/{added_year}"
                     "/{added_year_short}/{added_month}/{added_month_name}"
                     "/{added_month_name_short}/{added_day}/{asn}/{tags}"
-                    "/{tag_list}/",
+                    "/{tag_list}/{owner_username}/{original_name}/{dock_pk}/",
                 },
             ),
             content_type="application/json",
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] 'set' object has no attribute 'extend' when consuming documents
### Description

Document consumption does not work on 1.17.4-dev through docker compose with the image pulled from ghcr.io/paperless-ngx/paperless-ngx:dev

The problem seems to be here https://github.com/paperless-ngx/paperless-ngx/blob/dev/src/documents/data_models.py

### Steps to reproduce

1. put document in the consume folder
2. wait for error to popup in file tasks.

### Webserver logs

```bash
[2023-10-24 17:20:44,008] [DEBUG] [paperless.management.consumer] Waiting for file /usr/src/paperless/consume/test.doc to remain unmodified

[2023-10-24 17:20:49,009] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/test.doc to the task queue.

[2023-10-24 17:53:50,975] [INFO] [celery.worker.strategy] Task documents.tasks.consume_file[e8b4cb0a-7efb-425d-b14a-694ba95772a4] received
[2023-10-24 17:53:52,410] [ERROR] [celery.app.trace] Task documents.tasks.consume_file[e8b4cb0a-7efb-425d-b14a-694ba95772a4] raised unexpected: AttributeError("'set' object has no attribute 'extend'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/tasks.py", line 160, in consume_file
    overrides.update(template_overrides)
  File "/usr/src/paperless/src/documents/data_models.py", line 56, in update
    self.tag_ids.extend(other.tag_ids)
    ^^^^^^^^^^^^^^^^^^^
AttributeError: 'set' object has no attribute 'extend'
```


### Browser logs

_No response_

### Paperless-ngx version

1.17.4-dev

### Host OS

Synology NAS

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Please provide an actual traceback.
> Please provide an actual traceback.

Thats all the logs report in the section from the UI. I've updated the post with the docker console ones.

Resolving patch:
diff --git a/src/documents/management/commands/document_consumer.py b/src/documents/management/commands/document_consumer.py
--- a/src/documents/management/commands/document_consumer.py
+++ b/src/documents/management/commands/document_consumer.py
@@ -32,7 +32,7 @@
 logger = logging.getLogger("paperless.management.consumer")
 
 
-def _tags_from_path(filepath) -> set[Tag]:
+def _tags_from_path(filepath) -> list[int]:
     """
     Walk up the directory tree from filepath to CONSUMPTION_DIR
     and get or create Tag IDs for every directory.
@@ -47,7 +47,7 @@ def _tags_from_path(filepath) -> set[Tag]:
             Tag.objects.get_or_create(name__iexact=part, defaults={"name": part})[0].pk,
         )
 
-    return tag_ids
+    return list(tag_ids)
 
 
 def _is_ignored(filepath: str) -> bool:


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] Unable to delete notes in 2.00 beta rc1
### Description

Error delete notes in 2.00 beta rc1


### Steps to reproduce

Existing or newly created notes cannot be deleted
Newly created note overwrites existing old note

### Webserver logs

```bash
{"headers":{"normalizedNames":{},"lazyUpdate":null},"status":404,"statusText":"Not Found","url":"http://192.168.0.110:8777/api/documents/1812/notes/?id=421","ok":false,"name":"HttpErrorResponse","message":"Http failure response for http://192.168.0.110:8777/api/documents/1812/notes/?id=421: 404 Not Found","error":{"detail":"Nicht gefunden."}}
```


### Browser logs

_No response_

### Paperless-ngx version

2.00 beta rc1

### Host OS

Synology

### Installation method

Docker - official image

### Browser

Firefox

### Configuration changes

_No response_

### Other

_No response_


Discussion:
That's not actually a server log.  Please provide those.
I can reproduce this, I'll sort it out. Thanks for the report
Looks like maybe a frontend thing?  The backend (via admin) removes the note, but it hangs around in the UI

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -583,7 +583,7 @@ def notes(self, request, pk=None):
 
             from documents import index
 
-            index.add_or_update_document(self.get_object())
+            index.add_or_update_document(doc)
 
             return Response(self.getNotes(doc))
 


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] /api/remote_version/ returns version 0.0.0
### Description

After upgrading to v2.0.1, the endpoint /api/remote_version/ does not return the expected version code (2.0.1). 

The issue has also been reported by other users (see https://github.com/astubenbord/paperless-mobile/issues/314).

### Steps to reproduce

1. Go to <your-host>/api/remote_version/
2. Observe response


### Webserver logs

```bash
No notable logs (just startup script stuff)
```


### Browser logs

_No response_

### Paperless-ngx version

2.0.1

### Host OS

Ubuntu 20.04

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_


Discussion:
Addition: The ` X-Version` header seems to be set correctly.
Darn, looks like just a typo https://github.com/paperless-ngx/paperless-ngx/blob/6a78dde3e3b2302cb8e9cac26996561dc07476d4/src/documents/views.py#L1161

should be easy to fix

Resolving patch:
diff --git a/src/documents/views.py b/src/documents/views.py
--- a/src/documents/views.py
+++ b/src/documents/views.py
@@ -1158,8 +1158,8 @@ def get(self, request, format=None):
         current_version = packaging_version.parse(version.__full_version_str__)
         try:
             req = urllib.request.Request(
-                "https://api.github.com/repos/paperlessngx/"
-                "paperlessngx/releases/latest",
+                "https://api.github.com/repos/paperless-ngx/"
+                "paperless-ngx/releases/latest",
             )
             # Ensure a JSON response
             req.add_header("Accept", "application/json")


Test patch:
Repository: paperless-ngx/paperless-ngx

Issue:
[BUG] App Config does not enforce permissions
### Description

Since Paperless 2.3 or 2.4 there's a new menu item named "Configuration" where it's possible to override the logo and a few OCR settings. I'm using a group named `humans` for everyday operations. The group has mere "view" permissions for type "admin":

![image](https://github.com/paperless-ngx/paperless-ngx/assets/3238454/d500df7f-f95b-4d31-b91c-5df23acf6bd0)

And yet the users in that group can modify application configurations. I believe this is caused by the `/api/config/` endpoint not having authorization beyond `IsAuthenticated`:

https://github.com/paperless-ngx/paperless-ngx/blob/6d6650d5f6952f3129e9f4632cd149914b344767/src/paperless/views.py#L169

Removing the "view" permission bit, leaving the group without any for the "admin" type, only hides the UI elements. The underlying API is unaffected.

### Steps to reproduce

1. Authenticate with a user with no permissions for type "admin" other than "view" (the latter being optional if using the API directly).
1. Navigate to `/config` (Application Configuration).
1. Change values.
1. Save.

### Webserver logs

```bash
N/A
```


### Browser logs

_No response_

### Paperless-ngx version

2.4.0

### Host OS

Debian Linux

### Installation method

Docker - official image

### Browser

_No response_

### Configuration changes

_No response_

### Other

_No response_

### Please confirm the following

- [X] I believe this issue is a bug that affects all users of Paperless-ngx, not something specific to my installation.
- [X] I have already searched for relevant existing issues and discussions before opening this report.
- [X] I have updated the title field above with a concise description.


Discussion:

Resolving patch:
diff --git a/src/paperless/views.py b/src/paperless/views.py
--- a/src/paperless/views.py
+++ b/src/paperless/views.py
@@ -11,6 +11,7 @@
 from rest_framework.filters import OrderingFilter
 from rest_framework.generics import GenericAPIView
 from rest_framework.pagination import PageNumberPagination
+from rest_framework.permissions import DjangoObjectPermissions
 from rest_framework.permissions import IsAuthenticated
 from rest_framework.response import Response
 from rest_framework.viewsets import ModelViewSet
@@ -166,4 +167,4 @@ class ApplicationConfigurationViewSet(ModelViewSet):
     queryset = ApplicationConfiguration.objects
 
     serializer_class = ApplicationConfigurationSerializer
-    permission_classes = (IsAuthenticated,)
+    permission_classes = (IsAuthenticated, DjangoObjectPermissions)


Test patch: