internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: doccano/doccano

Issue:
Mutli-label text classification export issues: same classes but in different orders
How to reproduce the behaviour
---------
<!-- Before submitting an issue, make sure to check the docs and closed issues and FAQ to see if any of the solutions work for you. https://github.com/doccano/doccano/wiki/Frequently-Asked-Questions -->
We are two annotators on a multi-label classification project. When I export the annotations, for some examples, me and my co-annotator have put the same labels, but on the exported CSV, they do not appear in the same order:

Annotator 1:

| text | labels |
| example 1 | label1#label2#label3 |

Annotator 2:

| text | labels |
| example 1 | label2#label3#label1 |

As I try to use these CSVs for comparing our annotations, this brings more difficulty.

<!-- Include a code example or the steps that led to the problem. Please try to be as specific as possible. -->

Your Environment
---------
<!-- Include details of your environment.-->
*   Operating System: Debian
*   Python Version Used: Don't know, I pulled the latest version from Docker Hub
*   When you install doccano: 3 days ago
*   How did you install doccano (Heroku button etc): Docker



Discussion:

Resolving patch:
diff --git a/backend/api/views/download/writer.py b/backend/api/views/download/writer.py
--- a/backend/api/views/download/writer.py
+++ b/backend/api/views/download/writer.py
@@ -84,7 +84,7 @@ def create_line(self, record) -> Dict:
         return {
             'id': record.id,
             'data': record.data,
-            'label': '#'.join(record.label),
+            'label': '#'.join(sorted(record.label)),
             **record.metadata
         }
 
@@ -144,6 +144,7 @@ class FastTextWriter(LineWriter):
 
     def create_line(self, record):
         line = [f'__label__{label}' for label in record.label]
+        line.sort()
         line.append(record.data)
         line = ' '.join(line)
         return line


Test patch:
diff --git a/backend/api/tests/download/test_writer.py b/backend/api/tests/download/test_writer.py
--- a/backend/api/tests/download/test_writer.py
+++ b/backend/api/tests/download/test_writer.py
@@ -32,6 +32,16 @@ def test_create_line(self):
         }
         self.assertEqual(line, expected)
 
+    def test_label_order(self):
+        writer = CsvWriter('.')
+        record1 = Record(id=0, data='', label=['labelA', 'labelB'], user='', metadata={})
+        record2 = Record(id=0, data='', label=['labelB', 'labelA'], user='', metadata={})
+        line1 = writer.create_line(record1)
+        line2 = writer.create_line(record2)
+        expected = 'labelA#labelB'
+        self.assertEqual(line1['label'], expected)
+        self.assertEqual(line2['label'], expected)
+
     @patch('os.remove')
     @patch('zipfile.ZipFile')
     @patch('csv.DictWriter.writerow')
Repository: doccano/doccano

Issue:
Pagination of the project list
When fetching projects in the project list page, is it intentional that all projects are fetched at once even though there is pagination?

Endpoint of project list fetching: `/v1/projects`

When there are a lot of projects, it takes a long time to display them.

Your Environment
---------
doccano v1.5.5


Discussion:
Yes. It's intentional.
I thought we didn't join too many projects.

Do you think it should be paginated? It's relatively easy.
I think it should be paginated on fetching.
Even if there are fewer projects, pagination does not let performance lower.

Resolving patch:
diff --git a/backend/api/views/project.py b/backend/api/views/project.py
--- a/backend/api/views/project.py
+++ b/backend/api/views/project.py
@@ -1,5 +1,6 @@
 from django.conf import settings
-from rest_framework import generics, status
+from django_filters.rest_framework import DjangoFilterBackend
+from rest_framework import filters, generics, status
 from rest_framework.permissions import IsAdminUser, IsAuthenticated
 from rest_framework.response import Response
 
@@ -11,7 +12,8 @@
 
 class ProjectList(generics.ListCreateAPIView):
     serializer_class = ProjectPolymorphicSerializer
-    pagination_class = None
+    filter_backends = (DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter)
+    search_fields = ('name', 'description')
 
     def get_permissions(self):
         if self.request.method == 'GET':


Test patch:
diff --git a/backend/api/tests/api/test_project.py b/backend/api/tests/api/test_project.py
--- a/backend/api/tests/api/test_project.py
+++ b/backend/api/tests/api/test_project.py
@@ -15,13 +15,13 @@ def setUpTestData(cls):
     def test_return_projects_to_member(self):
         for member in self.project.members:
             response = self.assert_fetch(member, status.HTTP_200_OK)
-            project = response.data[0]
-            self.assertEqual(len(response.data), 1)
+            project = response.data['results'][0]
+            self.assertEqual(response.data['count'], 1)
             self.assertEqual(project['id'], self.project.item.id)
 
     def test_does_not_return_project_to_non_member(self):
         response = self.assert_fetch(self.non_member, status.HTTP_200_OK)
-        self.assertEqual(len(response.data), 0)
+        self.assertEqual(response.data['count'], 0)
 
 
 class TestProjectCreate(CRUDMixin):
Repository: doccano/doccano

Issue:
Database table for SpanType has invalid name 
How to reproduce the behaviour
---------
- Pull latest changes from master
- ./manage.py migrate
- ./api/migrations/0033_auto_20220127_0654.py will migrate the database table for `SpanType` to `label_types_spanType`
- Delete a project `Project.objects.first().delete()``

Exception:

<img width="511" alt="image" src="https://user-images.githubusercontent.com/6747788/152384221-a6a549b8-1cca-49c0-86e4-6a20f7d0a266.png">
 
The issue can be resolved by either renaming db table `label_types_spanType` to `label_types_spantype` or by explicitly setting `tb_table` for SpanType model like this: `db_table = "label_types_spanType"`

Your Environment
---------
*   Operating System: macOS Monterey, doccano is locally executed
*   Python Version Used: 3.9



Discussion:

Resolving patch:
diff --git a/backend/api/migrations/0033_auto_20220127_0654.py b/backend/api/migrations/0033_auto_20220127_0654.py
--- a/backend/api/migrations/0033_auto_20220127_0654.py
+++ b/backend/api/migrations/0033_auto_20220127_0654.py
@@ -34,7 +34,7 @@ class Migration(migrations.Migration):
                 ),
                 migrations.AlterModelTable(
                     name='SpanType',
-                    table='label_types_spanType'
+                    table='label_types_spantype'
                 )
             ]
         )


Test patch:
Repository: doccano/doccano

Issue:
How could I reduce number of workers?
Could I reduce the number_of_workers?
---------
I run the doccano in my machine use this code.
```
doccano init
doccano create user ***
doccano web server --port ***
```
And then I got this log: 
```
Booting worker with pid: 19
Booting worker with pid: 20
...
Booting worker with pid: 157
```
It run lots of worker and it took up a lot of memory. So, can I change the number_of_worker varlible. I saw the default number_of_worker= ``` multiprocessing.cpu_count()*2+1 ```. How could I change it?


Your Environment
---------
*   Operating System: Linux
*   Python Version Used: Python38
*   When you install doccano: 2021-11-30
*   How did you install doccano (Heroku button etc): pip install doccano



Discussion:
As I want to add a PR to solve this problem, I got this error:
```shell
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```
 😅

Resolving patch:
diff --git a/backend/cli.py b/backend/cli.py
--- a/backend/cli.py
+++ b/backend/cli.py
@@ -51,7 +51,7 @@ def load(self):
 
     options = {
         "bind": "%s:%s" % ("0.0.0.0", args.port),
-        "workers": number_of_workers(),
+        "workers": args.workers,
         "chdir": base,
         "capture_output": True,
         "loglevel": "debug",
@@ -134,6 +134,7 @@ def main():
     # Create a parser for web server.
     parser_server = subparsers.add_parser("webserver", help="see `webserver -h`")
     parser_server.add_argument("--port", type=int, default=8000, help="port number")
+    parser_server.add_argument("--workers", type=int, default=number_of_workers(), help="the number of workers")
     parser_server.set_defaults(handler=command_run_webserver)
 
     # Create a parser for task queue.


Test patch:
Repository: doccano/doccano

Issue:
Cannot access Django admin panel in a Heroku deployment
How to reproduce the behaviour
---------
The FAQ describes how to [create a user via the Django admin panel](https://github.com/doccano/doccano/blob/master/docs/faq.md#how-to-create-a-user) for a locally hosted Doccano. When run locally, I have no problem to reach the admin panel on `http://localhost:8000/admin/`, in Heroku however it is not working.

I have tried to reach it on
- `https://mydeployment.herokuapp.com/admin/`
- `https://mydeployment.herokuapp.com/admin/login`
- `https://mydeployment.herokuapp.com/admin/login/`
- `http://mydeployment.herokuapp.com/admin/`

Those urls all result in a `500 Internal Server Error`.
Am I missing something here, or is this perhaps a bug?

Your Environment
---------
<!-- Include details of your environment. -->

*   Operating System: -
*   Python Version Used: -
*   When did you install doccano: A few days ago
*   How did you install doccano (Heroku button etc): Heroku button



Discussion:
Having the exact same problem, and cannot see anything in the logs as it is an internal error
Same problem here!
I'm having the same issue. Has any1 here found a solution?
Same problem, does anybody have a solution?
@leugh I found a very simple solution.

Devs for doccano are either extremely busy or don't really care, so [lighttag](https://www.lighttag.io) is a great alternative, think there's a limit of around 5000 annotations before you have to pay but it works great for smaller datasets.  
same problem here
Probably need to set [heroku.py](https://github.com/doccano/doccano/blob/master/backend/config/settings/heroku.py) instead of [production.py](https://github.com/doccano/doccano/blob/master/backend/config/settings/production.py) in [Dockerfile.heroku](https://github.com/doccano/doccano/blob/master/docker/Dockerfile.heroku) and fix some settings.

Resolving patch:
diff --git a/backend/config/settings/heroku.py b/backend/config/settings/heroku.py
--- a/backend/config/settings/heroku.py
+++ b/backend/config/settings/heroku.py
@@ -2,4 +2,4 @@
 
 from .base import *  # noqa: F401,F403
 
-django_heroku.settings(locals(), test_runner=False)
+django_heroku.settings(locals(), test_runner=False, staticfiles=False)


Test patch:
Repository: doccano/doccano

Issue:
[Bug report] Static files are not copied on pip installation
How to reproduce the behaviour
---------
Seems like (some?) static files are not copied on pip installation.
For instance `http://site.com/favicon.ico` is available on Docker Compose installation. But it is 404'd on pip installation.

Your Environment
---------
<!-- Include details of your environment.-->
*   Operating System: CentOS 8.3
*   Python Version Used: 3.8.8
*   When you install doccano: v1.2.1
*   How did you install doccano (Heroku button etc): pip



Discussion:

Resolving patch:
diff --git a/backend/config/urls.py b/backend/config/urls.py
--- a/backend/config/urls.py
+++ b/backend/config/urls.py
@@ -15,6 +15,7 @@
 """
 import os
 import re
+from pathlib import Path
 
 from django.conf import settings
 from django.contrib import admin
@@ -36,6 +37,7 @@
 
 urlpatterns = []
 if settings.DEBUG or os.environ.get("STANDALONE", False):
+    static_dir = Path(__file__).resolve().parent.parent / "client" / "dist"
     # For showing images and audios in the case of pip and Docker.
     urlpatterns.append(
         re_path(
@@ -44,6 +46,8 @@
             {"document_root": settings.MEDIA_ROOT},
         )
     )
+    # For showing favicon on the case of pip and Docker.
+    urlpatterns.append(path("favicon.ico", serve, {"document_root": static_dir, "path": "favicon.ico"}))
 
 urlpatterns += [
     path("admin/", admin.site.urls),


Test patch:
Repository: doccano/doccano

Issue:
[Feature Request] Allowed to add more metadata for a project
Feature description
---------
currently we have many annotation projects in doccano.
However,  it is not easy to find the the right project. Because the information for a project is only its name.
- If the user could add more metadata for a project will be good. Such as the created data, created user, description. And all those metadata could be shown in project list page to help the user find the project.
- the metadata for a project could be modified. For example,  we created the project in a bad name such as "DocumentationClassification-1". And we can't change the name.
- some way to search the project or sort the project or filter the project? For example, sort  the project by creation date or only shown the project created by a user.



Discussion:

Resolving patch:
diff --git a/backend/projects/serializers.py b/backend/projects/serializers.py
--- a/backend/projects/serializers.py
+++ b/backend/projects/serializers.py
@@ -49,6 +49,13 @@ class Meta:
 
 class ProjectSerializer(serializers.ModelSerializer):
     tags = TagSerializer(many=True, required=False)
+    author = serializers.SerializerMethodField()
+
+    @classmethod
+    def get_author(cls, instance):
+        if instance.created_by:
+            return instance.created_by.username
+        return ""
 
     class Meta:
         model = Project
@@ -58,9 +65,10 @@ class Meta:
             "description",
             "guideline",
             "project_type",
+            "created_at",
             "updated_at",
             "random_order",
-            "created_by",
+            "author",
             "collaborative_annotation",
             "single_class_classification",
             "is_text_project",
@@ -71,7 +79,9 @@ class Meta:
             "tags",
         ]
         read_only_fields = (
+            "created_at",
             "updated_at",
+            "author",
             "is_text_project",
             "can_define_label",
             "can_define_relation",
diff --git a/backend/projects/views/project.py b/backend/projects/views/project.py
--- a/backend/projects/views/project.py
+++ b/backend/projects/views/project.py
@@ -13,6 +13,8 @@ class ProjectList(generics.ListCreateAPIView):
     serializer_class = ProjectPolymorphicSerializer
     filter_backends = (DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter)
     search_fields = ("name", "description")
+    ordering_fields = ["name", "created_at", "created_by", "project_type"]
+    ordering = ["-created_at"]
 
     def get_permissions(self):
         if self.request.method == "GET":


Test patch:
Repository: doccano/doccano

Issue:
CSRF
![image](https://user-images.githubusercontent.com/20540661/204228990-94d738c9-8bf7-41a9-8b81-d88acaa55148.png)



Discussion:
Would you write your environment? Thank you!
> Would you write your environment? Thank you!

environment:
centos
doccano:1.6.0
solved：https://blog.csdn.net/qq_23953717/article/details/128084659?csdn_share_tail=%7B%22type%22%3A%22blog%22%2C%22rType%22%3A%22article%22%2C%22rId%22%3A%22128084659%22%2C%22source%22%3A%22qq_23953717%22%7D
I would flag this as a duplicate of #2062 both for bug report and solution

Resolving patch:
diff --git a/backend/config/settings/base.py b/backend/config/settings/base.py
--- a/backend/config/settings/base.py
+++ b/backend/config/settings/base.py
@@ -230,8 +230,9 @@
 ALLOWED_HOSTS = ["*"]
 
 if DEBUG:
-    CORS_ORIGIN_WHITELIST = ("http://127.0.0.1:3000", "http://0.0.0.0:3000", "http://localhost:3000")
-    CSRF_TRUSTED_ORIGINS = CORS_ORIGIN_WHITELIST
+    CORS_ORIGIN_ALLOW_ALL = True
+    CSRF_TRUSTED_ORIGINS = ["http://127.0.0.1:3000", "http://0.0.0.0:3000", "http://localhost:3000"]
+    CSRF_TRUSTED_ORIGINS += env.list("CSRF_TRUSTED_ORIGINS", [])
 
 # Batch size for importing data
 IMPORT_BATCH_SIZE = env.int("IMPORT_BATCH_SIZE", 1000)
diff --git a/backend/config/settings/development.py b/backend/config/settings/development.py
--- a/backend/config/settings/development.py
+++ b/backend/config/settings/development.py
@@ -1,8 +1,6 @@
 from .base import *  # noqa: F403
 
 MIDDLEWARE.append("api.middleware.RangesMiddleware")  # noqa: F405
-CORS_ORIGIN_WHITELIST = ("http://127.0.0.1:3000", "http://0.0.0.0:3000", "http://localhost:3000")
-CSRF_TRUSTED_ORIGINS = CORS_ORIGIN_WHITELIST
 # LOGGING = {
 #     'version': 1,
 #     'handlers': {


Test patch:
Repository: doccano/doccano

Issue:
data task error
*   Operating System:ubuntu22
*   Python Version Used:3.9
*   When you install doccano:2022-12-04
*   How did you install doccano (Heroku button etc):pip install doccano

<img width="732" alt="image" src="https://user-images.githubusercontent.com/43643599/205482588-64111fa6-23bc-4c19-ade7-a7fb4532b177.png">




Discussion:
Would you write your environment? Thank you!
same error with miniconda environment 

Resolving patch:
diff --git a/backend/cli.py b/backend/cli.py
--- a/backend/cli.py
+++ b/backend/cli.py
@@ -155,7 +155,7 @@ def main():
     # Create a parser for task queue.
     parser_queue = subparsers.add_parser("task", help="see `task -h`")
     parser_queue.add_argument("--concurrency", type=int, default=2, help="concurrency")
-    parser_queue.add_argument("--env_file", type=str, default="", help="read in a file of environment variables")
+    parser_queue.add_argument("--env_file", type=str, help="read in a file of environment variables")
     parser_queue.set_defaults(handler=command_run_task_queue)
 
     parser_flower = subparsers.add_parser("flower", help="see `flower -h`")


Test patch:
Repository: doccano/doccano

Issue:
Include Example number in the Project Comments view
Feature description
---------
On the backend API and in the dataset annotation interface, the Comments are associated with specific Examples. But in the Project Comments view, the Example association is unclear--all the comments are grouped together.

Can the Project Comments view tab be improved to detail Examples, maybe even sort or group by Example?

Thanks!


Discussion:
Would you write your environment? Thank you!
Using the Docker Compose setup, running on Mac OS X Catalina 10.15.7

Resolving patch:
diff --git a/backend/examples/views/comment.py b/backend/examples/views/comment.py
--- a/backend/examples/views/comment.py
+++ b/backend/examples/views/comment.py
@@ -12,9 +12,10 @@
 class CommentList(generics.ListCreateAPIView):
     permission_classes = [IsAuthenticated & IsProjectMember]
     serializer_class = CommentSerializer
-    filter_backends = (DjangoFilterBackend, filters.SearchFilter)
+    filter_backends = (DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter)
     filterset_fields = ["example"]
     search_fields = ("text",)
+    ordering_fields = ("created_at", "example")
 
     def get_queryset(self):
         queryset = Comment.objects.filter(example__project_id=self.kwargs["project_id"])


Test patch:
Repository: EleutherAI/gpt-neox

Issue:
Implement 1-Bit Adam
Integrate 1-bit Adam into our model. The DeepSpeed tutorial can be found [here](https://www.deepspeed.ai/tutorials/onebit-adam/)


Discussion:

Resolving patch:
diff --git a/train_pipeline.py b/train_pipeline.py
--- a/train_pipeline.py
+++ b/train_pipeline.py
@@ -1,16 +1,21 @@
+import argparse
+import json
 import random
+from collections import defaultdict
+import os
 import deepspeed
 import torch
 from torch.utils.data import DataLoader
 from tqdm.auto import trange
-import torch.distributed as distributed
 
-from gpt_neox import (GPTNeoX_Pipe, AutoregressiveWrapper, GPT2Dataset, extract_tarfile,
-                      prepare_optimizer_parameters, get_tokenizer, is_main, prepare_data)
+from gpt_neox import (GPTNeoX, AutoregressiveWrapper, TextSamplerDataset,
+                      cycle, prepare_optimizer_parameters, decode_tokens, prepare_data,
+                      GPTNeoX_Pipe)
+from gpt_neox.datasets import GPT2Dataset
+from gpt_neox.utils import is_main
+import gpt_neox
 
-from gpt_neox.utils import get_args, get_params
-
-import GPUtil
+WORLD_SIZE = os.getenv('WORLD_SIZE')
 
 # arguments
 train_args = get_args()


Test patch:
Repository: EleutherAI/gpt-neox

Issue:
'attention.bias' and 'attention.masked_bias' not in `hf_layer.state_dict()` when converting gpt-neox model to huggingface
**Describe the bug**
A clear and concise description of what the bug is.

I encounter the following error when I am converting GPTNeoX models to Huggingface using the `tools/convert_module_to_hf.py` script.

```
(gpt-neox) johnny@ink-lucy:~/gpt-neox$ bash haveibeentrainedon/wikitext/pilot/convert_to_hf.sh 
[2023-08-18 23:37:21,695] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
> building GPT2BPETokenizer tokenizer ...
 > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
Saving weights in fp16 precision...
  0%|                                                                                           | 0/24 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "./tools/convert_module_to_hf.py", line 307, in <module>
    hf_model = convert(args.input_dir, loaded_config, args.output_dir)
  File "./tools/convert_module_to_hf.py", line 230, in convert
    state_dict["attention.bias"] = hf_layer.state_dict()["attention.bias"]
KeyError: 'attention.bias'
``` 

**Expected behavior**
Successful conversion.

**Proposed solution**
If you comment out lines 230 and 231, the script will run through. From an eyeballing of the results, it doesn't seem like language modelling performance seriously degraded. Could this be some code that was supposed to be taken out?

**Additional context**
This is for a model trained with the config `configs/pythia/410m.yml`



Discussion:
Use `pip install transformers==4.30.2`

`transformers>=4.31.0`:
ipdb> hf_layer.state_dict().keys()
odict_keys(['input_layernorm.weight', 'input_layernorm.bias', 'post_attention_layernorm.weight', 'post_attention_layernorm.bias', 'attention.rotary_emb.inv_freq', 'attention.query_key_value.weight', 'attention.query_key_value.bias', 'attention.dense.weight', 'attention.dense.bias', 'mlp.dense_h_to_4h.weight', 'mlp.dense_h_to_4h.bias', 'mlp.dense_4h_to_h.weight', 'mlp.dense_4h_to_h.bias'])

`transformers<=4.30.2` :
ipdb> hf_layer.state_dict().keys()
odict_keys(['input_layernorm.weight', 'input_layernorm.bias', 'post_attention_layernorm.weight', 'post_attention_layernorm.bias', 'attention.bias', 'attention.masked_bias', 'attention.rotary_emb.inv_freq', 'attention.query_key_value.weight', 'attention.query_key_value.bias', 'attention.dense.weight', 'attention.dense.bias', 'mlp.dense_h_to_4h.weight', 'mlp.dense_h_to_4h.bias', 'mlp.dense_4h_to_h.weight', 'mlp.dense_4h_to_h.bias'])

You can find missing 'attention.bias', 'attention.masked_bias' before `transformers==4.30.2`.

Resolving patch:
diff --git a/tools/corpora.py b/tools/corpora.py
--- a/tools/corpora.py
+++ b/tools/corpora.py
@@ -290,7 +290,7 @@ class C4OpenWebText(DataDownloader):
 
 class Enwik8(DataDownloader):
     name = "enwik8"
-    urls = ["https://data.deepai.org/enwik8.zip"]
+    urls = ["http://mattmahoney.net/dc/enwik8.zip"]
 
 
 def maybe_download_gpt2_tokenizer_data(tokenizer_type, data_dir):


Test patch: