internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: doccano/doccano

Issue:
Signup verification email not received
How to reproduce the behaviour
---------
I setup the project using AWS one-click deployment button. Everything works fine, but when a new user sign ups, email verification is not received. I believe I have to set up a email host configurations in `settings.py`. How do I set it up as the project has already been deployed? Is it in the `/env.list` file, or the AWS one-click automatically does this?

```
# necessary for email verification of new accounts
EMAIL_USE_TLS = env.bool('EMAIL_USE_TLS', False)
EMAIL_HOST = env('EMAIL_HOST', None)
EMAIL_HOST_USER = env('EMAIL_HOST_USER', None)
EMAIL_HOST_PASSWORD = env('EMAIL_HOST_PASSWORD', None)
EMAIL_PORT = env.int('EMAIL_PORT', 587)
```
Your Environment
---------
<!-- Include details of your environment. -->

*   Operating System: AWS ubuntu
*   Python Version Used: 3.6
*   When you install doccano: Mar 30, 2020
*   How did you install doccano (Heroku button etc): AWS one-click deployment

---------
Also, when I deployed the project using docker-compose by pulling the github project, the project looks older and not as in demo http://doccano.herokuapp.com/. Why is that? Am I missing something here?




Discussion:
I am having the same issue when deploying doccano using the one-click button for Heroku. 

Any advice would be greatly helpful. 
@srbek-cmd I have figured out how to make email verification work. You have to add the Email host configurations in the `/env.list` file and rerun the doccano.

> Note: It will erase all the previous doccano containers and create a new one, so all your old data from doccano will be erased.

Eg:

/env.list
```
ADMIN=admin
EMAIL=admin@email.com
PASSWORD=password
DEBUG=False
SECRET_KEY=your_secret_key

EMAIL_USE_TLS=True
EMAIL_HOST=smtp.gmail.com
EMAIL_HOST_USER=email_host@gmail.com
EMAIL_HOST_PASSWORD=email_host_password
EMAIL_PORT=587
```

Then remove the docker containers and rebuild them using the new /env.list file.
```
sudo docker stop doccano
sudo docker rm doccano
sudo docker run -d --name doccano --env-file /env.list -p 80:8000 chakkiworks/doccano:latest
sudo docker exec doccano tools/create-admin.sh ${ADMIN} ${EMAIL} ${PASSWORD}
```


Resolving patch:
diff --git a/app/app/settings.py b/app/app/settings.py
--- a/app/app/settings.py
+++ b/app/app/settings.py
@@ -314,6 +314,7 @@
 EMAIL_HOST_USER = env('EMAIL_HOST_USER', None)
 EMAIL_HOST_PASSWORD = env('EMAIL_HOST_PASSWORD', None)
 EMAIL_PORT = env.int('EMAIL_PORT', 587)
+DEFAULT_FROM_EMAIL = env('DEFAULT_FROM_EMAIL', 'webmaster@localhost')
 
 if not EMAIL_HOST:
     EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'


Test patch:
Repository: doccano/doccano

Issue:
Can't change from "Checked" to "Not Checked"?
I have annotate following sentence in NER that annotate by others already.

Here is the sentence:
![before_checked](https://user-images.githubusercontent.com/10709391/69947375-7b50f600-1528-11ea-8702-dfee2871e9b7.png)

I clicked the top left button to see others annotation:
![checked](https://user-images.githubusercontent.com/10709391/69947376-7b50f600-1528-11ea-9b2b-1fbf78b24540.png)

However, when I don't want to see others annotation, I clicked the top left button and it dosen't recover:
![not checked again](https://user-images.githubusercontent.com/10709391/69947377-7b50f600-1528-11ea-9f05-8c26cc182c3f.png)




Discussion:

Resolving patch:
diff --git a/app/api/serializers.py b/app/api/serializers.py
--- a/app/api/serializers.py
+++ b/app/api/serializers.py
@@ -84,6 +84,13 @@ class Meta:
         fields = ('id', 'text', 'annotations', 'meta', 'annotation_approver')
 
 
+class ApproverSerializer(DocumentSerializer):
+
+    class Meta:
+        model = Document
+        fields = ('id', 'annotation_approver')
+
+
 class ProjectSerializer(serializers.ModelSerializer):
     current_users_role = serializers.SerializerMethodField()
 
diff --git a/app/api/views.py b/app/api/views.py
--- a/app/api/views.py
+++ b/app/api/views.py
@@ -20,7 +20,7 @@
 from .filters import DocumentFilter
 from .models import Project, Label, Document, RoleMapping, Role
 from .permissions import IsProjectAdmin, IsAnnotatorAndReadOnly, IsAnnotator, IsAnnotationApproverAndReadOnly, IsOwnAnnotation, IsAnnotationApprover
-from .serializers import ProjectSerializer, LabelSerializer, DocumentSerializer, UserSerializer
+from .serializers import ProjectSerializer, LabelSerializer, DocumentSerializer, UserSerializer, ApproverSerializer
 from .serializers import ProjectPolymorphicSerializer, RoleMappingSerializer, RoleSerializer
 from .utils import CSVParser, ExcelParser, JSONParser, PlainTextParser, CoNLLParser, AudioParser, iterable_to_io
 from .utils import JSONLRenderer
@@ -133,7 +133,7 @@ def post(self, request, *args, **kwargs):
         document = get_object_or_404(Document, pk=self.kwargs['doc_id'])
         document.annotations_approved_by = self.request.user if approved else None
         document.save()
-        return Response(DocumentSerializer(document).data)
+        return Response(ApproverSerializer(document).data)
 
 
 class LabelList(generics.ListCreateAPIView):


Test patch:
Repository: doccano/doccano

Issue:
Expose timestamps associated with annotations with the API
The `..Annotation` models currently stores the `created_at` and `updated_at` fields but they are not exposed by the API.

I'd like to propose exposing them through the API so that downstream analysis can be conducted using simple API calls, like those made using [`doccano-client`](https://github.com/doccano/doccano-client), for instance.


Discussion:

Resolving patch:
diff --git a/app/api/serializers.py b/app/api/serializers.py
--- a/app/api/serializers.py
+++ b/app/api/serializers.py
@@ -177,7 +177,7 @@ class DocumentAnnotationSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = DocumentAnnotation
-        fields = ('id', 'prob', 'label', 'user', 'document')
+        fields = ('id', 'prob', 'label', 'user', 'document', 'created_at', 'updated_at')
         read_only_fields = ('user', )
 
 
@@ -188,7 +188,7 @@ class SequenceAnnotationSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = SequenceAnnotation
-        fields = ('id', 'prob', 'label', 'start_offset', 'end_offset', 'user', 'document')
+        fields = ('id', 'prob', 'label', 'start_offset', 'end_offset', 'user', 'document', 'created_at', 'updated_at')
         read_only_fields = ('user',)
 
 
@@ -197,7 +197,7 @@ class Seq2seqAnnotationSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = Seq2seqAnnotation
-        fields = ('id', 'text', 'user', 'document', 'prob')
+        fields = ('id', 'text', 'user', 'document', 'prob', 'created_at', 'updated_at')
         read_only_fields = ('user',)
 
 
@@ -206,7 +206,7 @@ class Speech2textAnnotationSerializer(serializers.ModelSerializer):
 
     class Meta:
         model = Speech2textAnnotation
-        fields = ('id', 'prob', 'text', 'user', 'document')
+        fields = ('id', 'prob', 'text', 'user', 'document', 'created_at', 'updated_at')
         read_only_fields = ('user',)
 
 


Test patch:
Repository: doccano/doccano

Issue:
Duplication of the data occurs at the bottom of paging
How to reproduce the behaviour
---------
The data at bottom of every page is same, that also make reduplicate document when doing annotation.

![image](https://user-images.githubusercontent.com/26696711/97519948-21c1a780-19d5-11eb-9243-35a5534738c9.png)
![image](https://user-images.githubusercontent.com/26696711/97520098-69483380-19d5-11eb-8861-33e7415c4fca.png)
![image](https://user-images.githubusercontent.com/26696711/97520119-75cc8c00-19d5-11eb-868f-74e703cca7e4.png)



Your Environment
---------
*   Operating System: ubuntu 16
*   Python Version Used: python3.6
*   When you install doccano: 2020-10-26
*   How did you install doccano (Heroku button etc): github



Discussion:
Issue-Label Bot is automatically applying the label `bug` to this issue, with a confidence of 0.85. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 

 Links: [app homepage](https://github.com/marketplace/issue-label-bot), [dashboard](https://mlbot.net/data/doccano/doccano) and [code](https://github.com/hamelsmu/MLapp) for this bot.
@buriy @ramok @vinayaugustine @clarus 
Same issue. Besides, this problem only occurs in sequence labeling and randomize document order. I guess this bug is caused by a wrong randomization algorithm. Hope that will help.
related #933

Resolving patch:
diff --git a/app/api/views.py b/app/api/views.py
--- a/app/api/views.py
+++ b/app/api/views.py
@@ -1,5 +1,7 @@
 import collections
 import json
+import random
+
 from django.conf import settings
 from django.contrib.auth.models import User
 from django.db import transaction
@@ -106,7 +108,6 @@ def _get_user_completion_data(annotation_class, annotation_filter):
             set_user_data[ind_obj['user__username']].add(ind_obj['document__id'])
         return {i: len(set_user_data[i]) for i in set_user_data}
 
-
     def progress(self, project):
         docs = project.documents
         annotation_class = project.get_annotation_class()
@@ -171,7 +172,9 @@ def get_queryset(self):
 
         queryset = project.documents
         if project.randomize_document_order:
-            queryset = queryset.annotate(sort_id=F('id') % self.request.user.id).order_by('sort_id')
+            random.seed(self.request.user.id)
+            value = random.randrange(2, 20)
+            queryset = queryset.annotate(sort_id=F('id') % value).order_by('sort_id', 'id')
         else:
             queryset = queryset.order_by('id')
 


Test patch:
Repository: doccano/doccano

Issue:
[Question] Docker deployment is obsolete in doccano 1.x.x?
Docker part [has been removed]( https://github.com/doccano/doccano/commit/1be0986c7a9faa5cf24d5d91900bb0d1058ba49f#diff-04c6e90faac2675aa89e2176d2eec7d8R64) from [ReadMe doocumentation](https://github.com/doccano/doccano/blob/master/README.md) some time ago.

In contrast, information about Docker deployment still stays in the [Getting Started documentation](https://github.com/doccano/doccano/blob/master/docs/getting-started.md#docker). (BTW, these documents are very different now, see #785).

Is Docker deployment is obsolete in doccano 1.x.x?

Also, we have https://hub.docker.com/u/doccano docker hub repository without mention in documentation.


Discussion:

Resolving patch:
diff --git a/app/app/settings.py b/app/app/settings.py
--- a/app/app/settings.py
+++ b/app/app/settings.py
@@ -92,7 +92,7 @@
 TEMPLATES = [
     {
         'BACKEND': 'django.template.backends.django.DjangoTemplates',
-        'DIRS': [path.join(BASE_DIR, 'server/templates'), path.join(BASE_DIR, 'authentification/templates')],
+        'DIRS': [path.join(BASE_DIR, 'client/dist')],
         'APP_DIRS': True,
         'OPTIONS': {
             'context_processors': [
@@ -118,12 +118,7 @@
 STATIC_ROOT = path.join(BASE_DIR, 'staticfiles')
 
 STATICFILES_DIRS = [
-    static_path
-    for static_path in (
-        path.join(BASE_DIR, 'server', 'static', 'assets'),
-        path.join(BASE_DIR, 'server', 'static', 'static'),
-    )
-    if path.isdir(static_path)
+    path.join(BASE_DIR, 'client/dist/static'),
 ]
 
 STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
diff --git a/app/app/urls.py b/app/app/urls.py
--- a/app/app/urls.py
+++ b/app/app/urls.py
@@ -15,12 +15,11 @@
 """
 from django.conf import settings
 from django.contrib import admin
-from django.urls import path, include
-from django.contrib.auth.views import PasswordResetView, LogoutView
+from django.urls import path, include, re_path
+from django.contrib.auth.views import TemplateView
 from drf_yasg.views import get_schema_view
 from drf_yasg import openapi
 
-from server.views import LoginView
 
 # TODO: adds AnnotationList and AnnotationDetail endpoint.
 schema_view = get_schema_view(
@@ -34,16 +33,12 @@
 )
 
 urlpatterns = [
-    path('', include('authentification.urls')),
-    path('', include('server.urls')),
     path('admin/', admin.site.urls),
     path('social/', include('social_django.urls')),
-    path('login/', LoginView.as_view(), name='login'),
-    path('logout/', LogoutView.as_view(), name='logout'),
-    path('password_reset/', PasswordResetView.as_view(), name='password_reset'),
     path('api-auth/', include('rest_framework.urls')),
     path('v1/', include('api.urls')),
     path('swagger/', schema_view.with_ui('swagger', cache_timeout=0), name='schema-swagger-ui'),
+    re_path('', TemplateView.as_view(template_name='index.html')),
 ]
 
 if 'cloud_browser' in settings.INSTALLED_APPS:


Test patch:
Repository: doccano/doccano

Issue:
Invalid syntax error when running the newly installed doccano
### System information
- **OS Platform and Distribution**: Fedora 32
- **Python version**: Python 3.8.6

### The problem
When opening using the docker-compose, the backend shows an error code about invalid syntax:
```
File "/src/app/app/settings.py", line 283
 if (spec := importlib.util.find_spec(name)) is not None:
          ^
SyntaxError: invalid syntax
```
this results in `doccano_backend_1 exited with code 1`

I also tried downloading it through pip in my windows pc, and it had the same result. 

When using the prod file on docker-compose, it also shows the following error
```
nginx_1     | 2020/12/10 20:27:46 [emerg] 1#1: host not found in upstream "backend" in /etc/nginx/conf.d/nginx.conf:15
nginx_1     | nginx: [emerg] host not found in upstream "backend" in /etc/nginx/conf.d/nginx.conf:15
doccano_nginx_1 exited with code 1
```
It's my first time using doccano and I may did something wrong, but I'm certain i followed the instructions correctly. 
Thank you for the help in advance!


Discussion:

Resolving patch:
diff --git a/app/app/settings.py b/app/app/settings.py
--- a/app/app/settings.py
+++ b/app/app/settings.py
@@ -280,7 +280,8 @@
 
 # dynamic import to avoid installing psycopg2 on pip installation.
 name = 'django_heroku'
-if (spec := importlib.util.find_spec(name)) is not None:
+spec = importlib.util.find_spec(name)
+if spec is not None:
     module = importlib.util.module_from_spec(spec)
     sys.modules[name] = module
     spec.loader.exec_module(module)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -16,9 +16,26 @@
 with io.open(os.path.join(here, 'README.md'), encoding='utf-8') as f:
     long_description = '\n' + f.read()
 
-# Todo: make a cleaned requirements.txt
-required = [line.rstrip() for line in io.open(os.path.join(here, 'app/requirements.txt'))
-            if not line.startswith(('psycopg2', 'django-heroku', 'django-mssql-backend'))]
+required = [
+    'apache-libcloud>=3.2.0',
+    'colour>=0.1.5',
+    'conllu>=4.2.2',
+    'dj-database-url>=0.5.0',
+    'django-cors-headers>=3.5.0',
+    'django-filter>=2.4.0',
+    'django-rest-polymorphic>=0.1.9',
+    'djangorestframework-csv>=2.1.0',
+    'djangorestframework-xml>=2.0.0',
+    'drf-yasg>=1.20.0',
+    'environs>=9.2.0',
+    'furl>=2.1.0',
+    'pyexcel>=0.6.6',
+    'pyexcel-xlsx>=0.6.0',
+    'python-jose>=3.2.0',
+    'seqeval>=1.2.2',
+    'social-auth-app-django>=4.0.0',
+    'whitenoise>=5.2.0'
+]
 
 setup(
     name=NAME,


Test patch:
Repository: doccano/doccano

Issue:
Feature Request: Bulk Delete
I want to clear all data but don't want delet the project. So i use the function in Dataset, i find something wrong.
1、Sometimes it doesn't show the remaining data unless i refresh the project. I want to delete 100 pieces of data at a time, so I have to choose 100 again.  It is inconvenient.
![image](https://user-images.githubusercontent.com/18391135/71570737-bb989980-2b11-11ea-8fd4-52548cb85e01.png)

2、So I hope there is a button to clear all the data.




Discussion:
It also would be nice to separate the imported datasets with labels. Then just one dataset could be deleted
+1 defo want this feat! thanks! 🤩

Resolving patch:
diff --git a/app/api/views.py b/app/api/views.py
--- a/app/api/views.py
+++ b/app/api/views.py
@@ -184,6 +184,12 @@ def perform_create(self, serializer):
         project = get_object_or_404(Project, pk=self.kwargs['project_id'])
         serializer.save(project=project)
 
+    def delete(self, request, *args, **kwargs):
+        project = get_object_or_404(Project, pk=self.kwargs['project_id'])
+        queryset = project.documents
+        queryset.all().delete()
+        return Response(status=status.HTTP_204_NO_CONTENT)
+
 
 class DocumentDetail(generics.RetrieveUpdateDestroyAPIView):
     queryset = Document.objects.all()
@@ -221,6 +227,11 @@ def create(self, request, *args, **kwargs):
     def perform_create(self, serializer):
         serializer.save(document_id=self.kwargs['doc_id'], user=self.request.user)
 
+    def delete(self, request, *args, **kwargs):
+        queryset = self.get_queryset()
+        queryset.all().delete()
+        return Response(status=status.HTTP_204_NO_CONTENT)
+
     @staticmethod
     def check_single_class_classification(project_id, doc_id, user):
         project = get_object_or_404(Project, pk=project_id)


Test patch:
Repository: doccano/doccano

Issue:
[Enhancement request] Meaningful error on labels naming conflict
Feature description
---------
Try rename a label to an existing name.

You get a 500 error.

Desired: a meaningful error.

Related: #601, #826.


Discussion:

Resolving patch:
diff --git a/app/api/exceptions.py b/app/api/exceptions.py
--- a/app/api/exceptions.py
+++ b/app/api/exceptions.py
@@ -34,3 +34,8 @@ class AWSTokenError(ValidationError):
 class SampleDataException(ValidationError):
     default_detail = 'The response is empty. Maybe the sample data is not appropriate.' \
                      'Please specify another sample data which returns at least one label.'
+
+
+class LabelValidationError(APIException):
+    status_code = status.HTTP_400_BAD_REQUEST
+    default_detail = 'You cannot create a label with same name or shortcut key.'
diff --git a/app/api/views/label.py b/app/api/views/label.py
--- a/app/api/views/label.py
+++ b/app/api/views/label.py
@@ -9,6 +9,7 @@
 from rest_framework.response import Response
 from rest_framework.views import APIView
 
+from ..exceptions import LabelValidationError
 from ..models import Label, Project
 from ..permissions import IsInProjectReadOnlyOrAdmin, IsProjectAdmin
 from ..serializers import LabelSerializer
@@ -27,6 +28,11 @@ def perform_create(self, serializer):
         project = get_object_or_404(Project, pk=self.kwargs['project_id'])
         serializer.save(project=project)
 
+    def delete(self, request, *args, **kwargs):
+        delete_ids = request.data['ids']
+        Label.objects.filter(pk__in=delete_ids).delete()
+        return Response(status=status.HTTP_204_NO_CONTENT)
+
 
 class LabelDetail(generics.RetrieveUpdateDestroyAPIView):
     queryset = Label.objects.all()
@@ -43,14 +49,14 @@ class LabelUploadAPI(APIView):
     def post(self, request, *args, **kwargs):
         if 'file' not in request.data:
             raise ParseError('Empty content')
-        labels = json.load(request.data['file'])
         project = get_object_or_404(Project, pk=kwargs['project_id'])
         try:
-            for label in labels:
-                serializer = LabelSerializer(data=label)
-                serializer.is_valid(raise_exception=True)
-                serializer.save(project=project)
+            labels = json.load(request.data['file'])
+            serializer = LabelSerializer(data=labels, many=True)
+            serializer.is_valid(raise_exception=True)
+            serializer.save(project=project)
             return Response(status=status.HTTP_201_CREATED)
+        except json.decoder.JSONDecodeError:
+            raise ParseError('The file format is invalid.')
         except IntegrityError:
-            content = {'error': 'IntegrityError: you cannot create a label with same name or shortkey.'}
-            return Response(content, status=status.HTTP_400_BAD_REQUEST)
+            raise LabelValidationError


Test patch:
Repository: doccano/doccano

Issue:
After Quick Deploy on Heroku, App crash due to sqlalchemy
How to reproduce the behaviour
---------
Start the quick deploy to heroku
Wait for successful build and deploy
Open app and wait for crash.

**Logs** 
`sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgres`
 `celery crashed`
`Process exited with status 12021-04-23T17:02:29.297622+00:00 heroku[web.1]: State changed from up to crashed`

Apparently versions of SQLAlchemy 1.4+ require postgresql:// and do not accept postgre:// anymore.

The DATABASE_URL start with postgres:// after deployment on heroku

Your Environment
---------
Installed with the deploy to heroku from doccano's home page, built from Dockerfile.
SQLAlchemy was 1.4.7



Discussion:
After forking and updating Pipfiles with SQLAlchemy==1.3.24, app is up and running on heroku.
SQLAlchemy related errors disappeared from logs.
Related:
- [Why is SQLAlchemy 1.4.x not connecting to Heroku Postgres?](https://help.heroku.com/ZKNTJQSK/why-is-sqlalchemy-1-4-x-not-connecting-to-heroku-postgres)

Resolving patch:
diff --git a/backend/app/settings.py b/backend/app/settings.py
--- a/backend/app/settings.py
+++ b/backend/app/settings.py
@@ -353,7 +353,12 @@
     CELERY_BROKER_URL = env('CELERY_BROKER_URL')
 except EnvError:
     try:
-        CELERY_BROKER_URL = 'sqla+{}'.format(env('DATABASE_URL'))
+        # quickfix for Heroku.
+        # See https://github.com/doccano/doccano/issues/1327.
+        uri = env('DATABASE_URL')
+        if uri.startswith('postgres://'):
+            uri = uri.replace('postgres://', 'postgresql://', 1)
+        CELERY_BROKER_URL = 'sqla+{}'.format(uri)
     except EnvError:
         CELERY_BROKER_URL = 'sqla+sqlite:///{}'.format(DATABASES['default']['NAME'])
 CELERY_ACCEPT_CONTENT = ['application/json']


Test patch: