internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: open-mmlab/mmocr

Issue:
Write image name to pickle file
Hi MMOCR team,
Thank you for this awesome framework. I have a task to get coordinate of bounding box from Textsnake model, so I use --out argument in test.py to export to a pickle file. But when I load this pickle, I just got ‘boundary_result’ and don't know this ‘boundary_result’ belongs to which image. How can I get the image to write to the pickle file? Thank you.


Discussion:

Resolving patch:
diff --git a/mmocr/models/textdet/dense_heads/head_mixin.py b/mmocr/models/textdet/dense_heads/head_mixin.py
--- a/mmocr/models/textdet/dense_heads/head_mixin.py
+++ b/mmocr/models/textdet/dense_heads/head_mixin.py
@@ -59,7 +59,9 @@ def get_boundary(self, score_maps, img_metas, rescale):
             boundaries = self.resize_boundary(
                 boundaries,
                 1.0 / self.downsample_ratio / img_metas[0]['scale_factor'])
-        results = dict(boundary_result=boundaries)
+        results = dict(
+            boundary_result=boundaries, filename=img_metas[0]['filename'])
+
         return results
 
     def loss(self, pred_maps, **kwargs):


Test patch:
diff --git a/tests/test_models/test_detector.py b/tests/test_models/test_detector.py
--- a/tests/test_models/test_detector.py
+++ b/tests/test_models/test_detector.py
@@ -380,7 +380,8 @@ def test_textsnake(cfg_file):
 
     one_meta = img_metas[0]
     result = detector.bbox_head.get_boundary(maps, [one_meta], False)
-    assert len(result) == 1
+    assert 'boundary_result' in result
+    assert 'filename' in result
 
     # Test show result
     results = {'boundary_result': [[0, 0, 1, 0, 1, 1, 0, 1, 0.9]]}
@@ -519,7 +520,7 @@ def test_drrg(cfg_file):
         detector.bbox_head.out_conv.bias.data.fill_(0.)
         outs = detector.bbox_head.single_test(maps)
         boundaries = detector.bbox_head.get_boundary(*outs, img_metas, True)
-    assert len(boundaries)
+    assert len(boundaries) == 1
 
     # Test show result
     results = {'boundary_result': [[0, 0, 1, 0, 1, 1, 0, 1, 0.9]]}
Repository: open-mmlab/mmocr

Issue:
MMOCR doesn't detect and recognize chinese characters.
I tried to detect and recognise Chinese characters, but it doesn't work. The output is just English characters.
I tried all combinations of supported detection models and recognition models.
In the paper, they describe the MMOCR can work on two languages: Eng + Chn, but what's wrong?




Discussion:
We have released SAR for Chinese recognition on https://mmocr.readthedocs.io/en/latest/textrecog_models.html#chinese-dataset. All other models were only trained on English characters for now.
I used this model: MMOCR(det='TextSnake', recog='SAR'). But it doesn't recognise Chinese. Recognised text is still English.
We haven't provided an out-of-the-box demo for Chinese OCR yet but will do it soon. For now, you need to specify the config (provided in `configs` folder) and model path (the model can be downloaded from the link) manually to run Chinese SAR from `ocr.py`.

Resolving patch:
diff --git a/mmocr/utils/ocr.py b/mmocr/utils/ocr.py
--- a/mmocr/utils/ocr.py
+++ b/mmocr/utils/ocr.py
@@ -273,6 +273,12 @@ def __init__(self,
                 'config': 'sar/sar_r31_parallel_decoder_academic.py',
                 'ckpt': 'sar/sar_r31_parallel_decoder_academic-dba3a4a3.pth'
             },
+            'SAR_CN': {
+                'config':
+                'sar/sar_r31_parallel_decoder_chinese.py',
+                'ckpt':
+                'sar/sar_r31_parallel_decoder_chineseocr_20210507-b4be8214.pth'
+            },
             'NRTR_1/16-1/8': {
                 'config': 'nrtr/nrtr_r31_1by16_1by8_academic.py',
                 'ckpt': 'nrtr/nrtr_r31_academic_20210406-954db95e.pth'


Test patch:
Repository: open-mmlab/mmocr

Issue:
AssertionError in tools/data/textdet/totaltext_converter.py
I have trouble when running totaltext_converter.py
```
python tools/data/textdet/totaltext_converter.py data/totaltext -o data/totaltext --split-list training test
Converting training into instances_training.json
Loaded 1255 images from data/totaltext/imgs/training
[>>>>>>>>>>>>>>>>>>>         ] 894/1255, 42.8 task/s, elapsed: 21s, ETA:     8sIt takes 20.903159141540527s to convert totaltext annotation
Traceback (most recent call last):
  File "tools/data/textdet/totaltext_converter.py", line 412, in <module>
    main()
  File "tools/data/textdet/totaltext_converter.py", line 407, in main
    image_infos = collect_annotations(files, 1)
  File "tools/data/textdet/totaltext_converter.py", line 69, in collect_annotations
    images = mmcv.track_progress(load_img_info, files)
  File "/home/lhb/anaconda3/envs/pytrackingLay/lib/python3.8/site-packages/mmcv/utils/progressbar.py", line 92, in track_progress
    results.append(func(task, **kwargs))
  File "tools/data/textdet/totaltext_converter.py", line 350, in load_img_info
    assert img.shape[0:2] == img_color.shape[0:2]
AssertionError
```
I searched in issues as much as possible but did not find a solution. So i come to ask for help. Thanks a lot.


Discussion:

Resolving patch:
diff --git a/tools/data/textdet/totaltext_converter.py b/tools/data/textdet/totaltext_converter.py
--- a/tools/data/textdet/totaltext_converter.py
+++ b/tools/data/textdet/totaltext_converter.py
@@ -344,10 +344,6 @@ def load_img_info(files):
     img_file, gt_file = files
     # read imgs with ignoring orientations
     img = mmcv.imread(img_file, 'unchanged')
-    # read imgs with orientations as dataloader does when training and testing
-    img_color = mmcv.imread(img_file, 'color')
-    # make sure imgs have no orientation info, or annotation gt is wrong.
-    assert img.shape[0:2] == img_color.shape[0:2]
 
     split_name = osp.basename(osp.dirname(img_file))
     img_info = dict(


Test patch:
Repository: open-mmlab/mmocr

Issue:
locals() should not be modified
```python
args = locals()
[args.pop(x, None) for x in ['kwargs', 'self']]
```
https://github.com/open-mmlab/mmocr/blob/b04775fd78ac89e32d38ef6fbd5493dedbfd76f4/mmocr/utils/ocr.py#L414


Discussion:

Resolving patch:
diff --git a/mmocr/utils/ocr.py b/mmocr/utils/ocr.py
--- a/mmocr/utils/ocr.py
+++ b/mmocr/utils/ocr.py
@@ -411,7 +411,7 @@ def readtext(self,
                  merge=False,
                  merge_xdist=20,
                  **kwargs):
-        args = locals()
+        args = locals().copy()
         [args.pop(x, None) for x in ['kwargs', 'self']]
         args = Namespace(**args)
 


Test patch:
Repository: open-mmlab/mmocr

Issue:
An error occurred while trying to generate a totaltext gt
Thanks for your error report and we appreciate it a lot.


**Describe the bug**
Traceback (most recent call last):
  File "tools/data/textdet/totaltext_converter.py", line 410, in <module>
    main()
  File "tools/data/textdet/totaltext_converter.py", line 405, in main
    image_infos = collect_annotations(files, nproc=args.nproc)
  File "tools/data/textdet/totaltext_converter.py", line 69, in collect_annotations
    images = mmcv.track_progress(load_img_info, files)
  File "/home/yj/anaconda3/envs/pytorch/lib/python3.7/site-packages/mmcv/utils/progressbar.py", line 92, in track_progress
    results.append(func(task, **kwargs))
  File "tools/data/textdet/totaltext_converter.py", line 362, in load_img_info
    img_info = load_txt_info(gt_file, img_info)
  File "tools/data/textdet/totaltext_converter.py", line 262, in load_txt_info
    contours, words = get_contours_txt(gt_file)
  File "tools/data/textdet/totaltext_converter.py", line 237, in get_contours_txt
    contours, words = process_line(complete_line, contours, words)
  File "tools/data/textdet/totaltext_converter.py", line 181, in process_line
    ann_dict = yaml.load(ann_dict)
TypeError: load() missing 1 required positional argument: 'Loader'

**Reproduction**

1. What command or script did you run?

```none
python tools/data/textdet/totaltext_converter.py ./data/totaltext -o ./data/totaltext --split-list training test
```

2. Did you make any modifications on the code or config? Did you understand what you have modified?
I didn't make any changes(mmocr 0.4.0)
3. What dataset did you use?
totaltext



Discussion:

Resolving patch:
diff --git a/tools/data/textdet/totaltext_converter.py b/tools/data/textdet/totaltext_converter.py
--- a/tools/data/textdet/totaltext_converter.py
+++ b/tools/data/textdet/totaltext_converter.py
@@ -180,7 +180,7 @@ def process_line(line, contours, words):
     ann_dict = re.sub('([0-9]) +([ 0-9])', r'\1,\2', ann_dict)
     ann_dict = re.sub('([0-9]) -([0-9])', r'\1,-\2', ann_dict)
     ann_dict = ann_dict.replace("[u',']", "[u'#']")
-    ann_dict = yaml.load(ann_dict)
+    ann_dict = yaml.safe_load(ann_dict)
 
     X = np.array([ann_dict['x']])
     Y = np.array([ann_dict['y']])
diff --git a/tools/data/textrecog/totaltext_converter.py b/tools/data/textrecog/totaltext_converter.py
--- a/tools/data/textrecog/totaltext_converter.py
+++ b/tools/data/textrecog/totaltext_converter.py
@@ -169,7 +169,7 @@ def process_line(line, contours, words):
     ann_dict = re.sub('([0-9]) +([ 0-9])', r'\1,\2', ann_dict)
     ann_dict = re.sub('([0-9]) -([0-9])', r'\1,-\2', ann_dict)
     ann_dict = ann_dict.replace("[u',']", "[u'#']")
-    ann_dict = yaml.load(ann_dict)
+    ann_dict = yaml.safe_load(ann_dict)
 
     X = np.array([ann_dict['x']])
     Y = np.array([ann_dict['y']])


Test patch:
Repository: open-mmlab/mmocr

Issue:
[Bug] OCR1.x dataset prepare don't support textspotting
### Prerequisite

- [X] I have searched [Issues](https://github.com/open-mmlab/mmocr/issues) and [Discussions](https://github.com/open-mmlab/mmocr/discussions) but cannot get the expected help.
- [X] The bug has not been fixed in the [latest version (0.x)](https://github.com/open-mmlab/mmocr) or [latest version (1.x)](https://github.com/open-mmlab/mmocr/tree/dev-1.x).

### Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

### Branch

1.x branch https://github.com/open-mmlab/mmocr/tree/dev-1.x

### Environment

mmocr==1.0.0rc3
openmim==0.3.2


ps:  python mmocr/utils/collect_env.py will get error in 1.x, so I just paste important pkg version

### Reproduces the problem - code sample

`python tools/dataset_converters/prepare_dataset.py icdar2015 --task textspotting`


### Reproduces the problem - command or script

python tools/dataset_converters/prepare_dataset.py icdar2015 --task textspotting

### Reproduces the problem - error message

FileNotFoundError: [Errno 2] No such file or directory: 'configs/textspotting/_base_/datasets/icdar2015.py'

### Additional information

and when download icdar2015,  there is no URL in textspotting.py


Discussion:

Resolving patch:
diff --git a/mmocr/datasets/preparers/data_converter.py b/mmocr/datasets/preparers/data_converter.py
--- a/mmocr/datasets/preparers/data_converter.py
+++ b/mmocr/datasets/preparers/data_converter.py
@@ -127,6 +127,7 @@ def generate_dataset_config(self, dataset_config: Dict) -> None:
                     break
                 if c.lower() == 'n':
                     return
+        mkdir_or_exist(osp.dirname(cfg_path))
         with open(cfg_path, 'w') as f:
             f.write(
                 f'{self.dataset_name}_{self.task}_data_root = \'{self.data_root}\'\n'  # noqa: E501


Test patch:
Repository: nextcloud/appstore

Issue:
add description field to category model
I think a description of the category would be a nice addition.

Thoughts?

@BernhardPosselt @LukasReschke 



Discussion:
Good idea :)

has to be l10ned


Resolving patch:
diff --git a/nextcloudappstore/core/migrations/0001_initial.py b/nextcloudappstore/core/migrations/0001_initial.py
--- a/nextcloudappstore/core/migrations/0001_initial.py
+++ b/nextcloudappstore/core/migrations/0001_initial.py
@@ -218,6 +218,12 @@ class Migration(migrations.Migration):
             name='owner',
             field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='owned_apps', to=settings.AUTH_USER_MODEL, verbose_name='App owner'),
         ),
+        migrations.AddField(
+            model_name='categorytranslation',
+            name='description',
+            field=models.TextField(default='This is a default description', help_text='Will be rendered as Markdown', verbose_name='Description'),
+            preserve_default=False,
+        ),
         migrations.AlterUniqueTogether(
             name='categorytranslation',
             unique_together=set([('language_code', 'master')]),
diff --git a/nextcloudappstore/core/models.py b/nextcloudappstore/core/models.py
--- a/nextcloudappstore/core/models.py
+++ b/nextcloudappstore/core/models.py
@@ -160,7 +160,10 @@ class Category(TranslatableModel):
     translations = TranslatedFields(
         name=models.CharField(max_length=128, help_text=_(
             'Category name which will be presented to the user'),
-                              verbose_name=_('Name'))
+                              verbose_name=_('Name')),
+        description=models.TextField(verbose_name=_('Description'),
+                                     help_text=_(
+                                         'Will be rendered as Markdown'))
     )
 
     class Meta:


Test patch:
Repository: nextcloud/appstore

Issue:
API routes should be enabled for CORS
Only API routes should be whitelisted for CORS

We need to make sure that CORS and session auth are mutually exclusive

The solution is probably to integrate https://github.com/ottoyiu/django-cors-headers/



Discussion:

Resolving patch:
diff --git a/nextcloudappstore/settings.py b/nextcloudappstore/settings.py
--- a/nextcloudappstore/settings.py
+++ b/nextcloudappstore/settings.py
@@ -21,28 +21,29 @@
 # Application definition
 
 INSTALLED_APPS = [
+    'nextcloudappstore.core.apps.CoreConfig',
     'parler',
+    'captcha',
     'rest_framework',
+    'corsheaders',
+    'allauth',
+    'allauth.account',
+    'allauth.socialaccount',
+    'allauth.socialaccount.providers.github',
+    'allauth.socialaccount.providers.bitbucket',
     'django.contrib.admin',
     'django.contrib.auth',
     'django.contrib.contenttypes',
     'django.contrib.sessions',
     'django.contrib.messages',
-    # The Django sites framework is required by allauth
     'django.contrib.sites',
     'django.contrib.staticfiles',
-    'captcha',
-    'nextcloudappstore.core.apps.CoreConfig',
-    'allauth',
-    'allauth.account',
-    'allauth.socialaccount',
-    'allauth.socialaccount.providers.github',
-    'allauth.socialaccount.providers.bitbucket',
 ]
 
 MIDDLEWARE_CLASSES = [
     'django.middleware.security.SecurityMiddleware',
     'django.contrib.sessions.middleware.SessionMiddleware',
+    'corsheaders.middleware.CorsMiddleware',
     'django.middleware.common.CommonMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
@@ -171,6 +172,22 @@
 # Default security settings
 SECURE_BROWSER_XSS_FILTER = True
 SECURE_CONTENT_TYPE_NOSNIFF = True
+CORS_ORIGIN_ALLOW_ALL = True
+CORS_URLS_REGEX = r'^/api/.*$'
+CORS_ALLOW_HEADERS = (
+    'x-requested-with',
+    'content-type',
+    'accept',
+    'origin',
+    'authorization',
+    'x-csrftoken',
+    'if-none-match',
+)
+CORS_EXPOSE_HEADERS = (
+    'etag',
+    'x-content-type-options',
+    'content-type',
+)
 
 # use modern no Captcha reCaptcha
 NOCAPTCHA = True


Test patch:
Repository: nextcloud/appstore

Issue:
Test if zip bombs are possible
We should know if zip bombs are possible currently



Discussion:

Resolving patch:
diff --git a/nextcloudappstore/core/api/v1/release/parser.py b/nextcloudappstore/core/api/v1/release/parser.py
--- a/nextcloudappstore/core/api/v1/release/parser.py
+++ b/nextcloudappstore/core/api/v1/release/parser.py
@@ -77,20 +77,44 @@ def _parse_archive(self, tar: Any) -> Tuple[str, str]:
 
             for possible_link in possible_links:
                 if possible_link.issym() or possible_link.islnk():
-                    msg = 'Symlinks and hard links can not be used for %s' %\
+                    msg = 'Symlinks and hard links can not be used for %s' % \
                           possible_link
                     raise InvalidAppPackageStructureException(msg)
-
-            if info_member.size > self.config.max_info_size:
-                msg = '%s was bigger than allowed %i bytes' % (
-                    info_path, self.config.max_info_size)
-                raise MaxSizeAppMetadataXmlException(msg)
             info_file = tar.extractfile(info_member)
-            return info_file.read().decode('utf-8'), app_id
+            contents = self._stream_read_file(info_file,
+                                              self.config.max_info_size)
+            return contents, app_id
         except KeyError:
             msg = 'Could not find %s file inside the archive' % info_path
             raise InvalidAppPackageStructureException(msg)
 
+    def _stream_read_file(self, info_file: Any, max_info_size: int) -> str:
+        """
+        Instead of reading everything in one go which is vulnerable to
+        zip bombs, stream and accumulate the bytes
+        :argument info_file: buffered io reader
+        :argument max_info_size: maximum file size in bytes
+        :raises MaxSizeAppMetadataXmlException if the maximum size was reached
+        :return: the parsed info.xml
+        """
+        # FIXME: If someone finds a less ugly version, please feel free to
+        #        improve it
+        size = 0
+        result = b''
+        while True:
+            size += 1024
+            if size > max_info_size:
+                msg = 'info.xml was bigger than allowed %i bytes' % \
+                      max_info_size
+                raise MaxSizeAppMetadataXmlException(msg)
+
+            chunk = info_file.read(1024)
+            if not chunk:
+                break
+            result += chunk
+
+        return result.decode('utf-8')
+
 
 def element_to_dict(element: Any) -> Dict:
     type = element.get('type')


Test patch:
Repository: nextcloud/appstore

Issue:
After clicking confirm button I got a 404
- click the confirm link in the email
- click the button on that page
- getting redirected to https://.../accounts/login/ instead of https://.../login/ which is not available

cc @BernhardPosselt @adsworth 



Discussion:
I'll look into it tomorrow.


Resolving patch:
diff --git a/nextcloudappstore/settings/base.py b/nextcloudappstore/settings/base.py
--- a/nextcloudappstore/settings/base.py
+++ b/nextcloudappstore/settings/base.py
@@ -188,3 +188,4 @@
 NOCAPTCHA = True
 
 LOGIN_REDIRECT_URL = 'home'
+LOGIN_URL = 'account_login'


Test patch:
Repository: nextcloud/appstore

Issue:
Nightly support
- nightlies don't have a separate version number but a flag:

```
curl -X POST -u "user:password" http://localhost:8000/api/v1/apps/releases -H "Content-Type: application/json" -d '{"download":"https://example.com/release.tar.gz", "nightly":true }'
```
- this is also listed in the "get all apps" API with a `nightly: true` attribute https://nextcloudappstore.readthedocs.io/en/latest/restapi.html#get-all-apps-and-releases
- upload of a new nightly will delete the previous one for that app
- this allows to upgrade to a nightly (needs to be invoked by the admin and can be undone -> next regular release of the app will be installed)



Discussion:

Resolving patch:
diff --git a/nextcloudappstore/core/api/v1/urls.py b/nextcloudappstore/core/api/v1/urls.py
--- a/nextcloudappstore/core/api/v1/urls.py
+++ b/nextcloudappstore/core/api/v1/urls.py
@@ -9,7 +9,8 @@
     url(r'^apps/releases/?$', AppReleases.as_view(),
         name='app-release-create'),
     url(r'^apps/(?P<pk>[a-z_]+)/?$', Apps.as_view(), name='app-delete'),
-    url(r'^apps/(?P<app>[a-z_]+)/releases/(?P<version>\d+\.\d+\.\d+)/?$',
+    url(r'^apps/(?P<app>[a-z_]+)/releases/(?P<version>\d+\.\d+\.\d+'
+        r'(?:-nightly)?)/?$',
         AppReleases.as_view(), name='app-release-delete'),
     url(r'^categories.json$',
         etag(category_api_etag)(Categories.as_view()), name='categories'),


Test patch:
Repository: nextcloud/appstore

Issue:
Email confirmation email should be improved
I couldn't find the text for the email, so I suspect it's from some library.

Here is the content:

---

Hello from **apps.nextcloud.com**!

You're receiving this e-mail because user oparoz at apps.nextcloud.com has given **yours as an e-mail address to connect their account**.

To confirm this is correct, go to https://apps.nextcloud.com/confirm-email/Mzc:1bZksL:Y8YI3zMQ0fOllevi3VhZ-dmiSMU/

Thank you from **apps.nextcloud.com**!
**apps.nextcloud.com**

---

I've highlighted what should be altered.



Discussion:
I think it's that one https://github.com/pennersr/django-allauth/blob/master/allauth/templates/account/email/email_confirmation_message.txt

To override it, copy the template into https://github.com/nextcloud/appstore/tree/master/nextcloudappstore/core/templates under the same directory and filename (/nextcloudappstore/core/templates/account/email/email_confirmation_message.txt) and change the bits that you want to improve.

PR welcome

The problem with apps.nextcloud.com showing up everywhere is a config issue. We'll just have to remember to set the variable correctly when going into production.

Fix for the content itself is at: https://github.com/nextcloud/appstore/pull/179


Resolving patch:
diff --git a/nextcloudappstore/core/management/commands/setupsocial.py b/nextcloudappstore/core/management/commands/setupsocial.py
--- a/nextcloudappstore/core/management/commands/setupsocial.py
+++ b/nextcloudappstore/core/management/commands/setupsocial.py
@@ -21,7 +21,7 @@ def handle(self, *args, **options):
         # set up site which is required for social login
         site = Site.objects.all()[0]
         site.domain = options['domain']
-        site.name = options['domain']
+        site.name = 'Nextcloud App Store'
         site.save()
         # set up github
         app, created = SocialApp.objects.get_or_create(provider='github')


Test patch: