internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dmlc/gluon-nlp

Issue:
'name' is an undefined name twice in evaluation.py
An _undefined name_ may cause NameError to be raised at runtime.

flake8 testing of https://github.com/dmlc/gluon-nlp on Python 3.6.3

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
../gluonnlp/embedding/evaluation.py:380:67: F821 undefined name 'name'
                '{} is not a WordEmbeddingAnalogyFunction'.format(name))
                                                                  ^
./gluonnlp/embedding/evaluation.py:439:67: F821 undefined name 'name'
                '{} is not a WordEmbeddingAnalogyFunction'.format(name))
                                                                  ^
```


Discussion:

Resolving patch:
diff --git a/gluonnlp/embedding/evaluation.py b/gluonnlp/embedding/evaluation.py
--- a/gluonnlp/embedding/evaluation.py
+++ b/gluonnlp/embedding/evaluation.py
@@ -377,7 +377,8 @@ def __init__(self, idx_to_vec, similarity_function='CosineSimilarity',
 
         if not isinstance(self.similarity, WordEmbeddingSimilarityFunction):
             raise RuntimeError(
-                '{} is not a WordEmbeddingAnalogyFunction'.format(name))
+                '{} is not a WordEmbeddingAnalogyFunction'.format(
+                    self.similarity.__class__.__name__))
 
     def hybrid_forward(self, F, words1, words2, weight):  # pylint: disable=arguments-differ
         """Predict the similarity of words1 and words2.
@@ -436,7 +437,8 @@ def __init__(self, idx_to_vec, analogy_function='ThreeCosMul', k=1,
 
         if not isinstance(self.analogy, WordEmbeddingAnalogyFunction):
             raise RuntimeError(
-                '{} is not a WordEmbeddingAnalogyFunction'.format(name))
+                '{} is not a WordEmbeddingAnalogyFunction'.format(
+                    self.analogy.__class__.__name__))
 
     def forward(self, words1, words2, words3):  # pylint: disable=arguments-differ
         """Implement forward computation.


Test patch:
Repository: dmlc/gluon-nlp

Issue:
Add pre-trained source cc.en.300 for token embedding fasttext
FaceBook Research recently released the English source files.
https://github.com/facebookresearch/fastText/commit/5bf8b4c615b6308d76ad39a5a50fa6c4174113ea
Could those be added to Gluon-NLP as well?



Discussion:
Yes, I can add the English vectors and crawl-300d-2M-subword.zip
Thank you very much! That will be greatly appreciated.
Also, at the moment it's not possible to use _wiki-news-300d-1M-subword.vec.zip_ as source with _load_ngrams=True_
Do I understand it correctly that this should be possible since it has been trained with subword information?
on english, i have observed wiki.en.vec to be much superior in performance over the common crawl embeddings, for other languages that arent data rich, the common crawl ones are doing good.

https://arxiv.org/pdf/1802.06893.pdf

Inline with what the common-crawl embeddings paper suggests
@KurtJanssensCRW unfortunately no. While those vectors were supposedly trained with subword information, the subword part of the embeddings was not published.

If you find the subword part (included in the `.bin` file), we'd be happy to add it. 
@szha could you copy the following files to the GluonNLP bucket

https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-e6b07293f.bin
https://lllausen-data.s3.amazonaws.com/gluonnlp/crawl-300d-2M-subword-927782c8e.npz

Thanks!
@leezu yes, done
@KurtJanssensCRW unlike wiki-news-300d-1M-subword.vec.zip, for crawl-300d-2M-subword the ngram vectors were published and can be used via `load_ngrams=True`.
Thank you for your effort!
I have submitted an issue to fastText to add the bin file for the English wikipedia embedding (https://github.com/facebookresearch/fastText/issues/656)

I noticed that you added crawl-300d-2M-subword but not cc.en.300 in https://github.com/dmlc/gluon-nlp/pull/336
Would it be possible to add these as well or are they inferior?

Many thanks,
Kurt
FacebookResearch confirmed that the bin file for English wikipedia embeddings is avalable:
https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki-news-300d-1M-subword.bin.zip
https://github.com/facebookresearch/fastText/issues/656
Would it be possible to add the bin files for the multi-language embeddings as well?
https://fasttext.cc/docs/en/crawl-vectors.html
Many thanks!

Resolving patch:
diff --git a/gluonnlp/_constants.py b/gluonnlp/_constants.py
--- a/gluonnlp/_constants.py
+++ b/gluonnlp/_constants.py
@@ -563,6 +563,8 @@
 
 
 FAST_TEXT_BIN_SHA1 = {
+    'wiki-news-300d-1M-subword': ('wiki-news-300d-1M-subword-c8853bda.bin',
+                                  'c8853bdae00318097b6337c4631d342879d6b18c'),
     'crawl-300d-2M-subword': ('crawl-300d-2M-subword-e6b07293f.bin',
                               'e6b07293f7b0095e3c72c2a12bc09464b69444b0'),
     'cc.af.300': ('cc.af.300-33115ff8.bin', '33115ff8e4c8f439757c819399177f1f58f07f12'),


Test patch:
Repository: dmlc/gluon-nlp

Issue:
Find a Bug in pretrain script of bert
https://github.com/dmlc/gluon-nlp/blob/master/scripts/bert/run_pretraining.py

line 296 and 297,throw a exception：


```
Traceback (most recent call last):
  File "/home/xzc/gluon-nlp/scripts/bert/run_pretraining.py", line 477, in <module>
    evaluate(data_eval, model, nsp_loss, mlm_loss, len(vocabulary), ctx)
  File "/home/xzc/gluon-nlp/scripts/bert/run_pretraining.py", line 297, in evaluate
    mlm_metric.reset_running()
AttributeError: 'MaskedAccuracy' object has no attribute 'reset_running'
```


Discussion:
@eric-haibin-lin I just checked the class `MaskedAccuracy`, there is no function called `reset_running`.
Sorry it should be reset_local instead. I'll make a PR with the fix

Resolving patch:
diff --git a/scripts/bert/run_pretraining.py b/scripts/bert/run_pretraining.py
--- a/scripts/bert/run_pretraining.py
+++ b/scripts/bert/run_pretraining.py
@@ -293,8 +293,8 @@ def evaluate(data_eval, model, nsp_loss, mlm_loss, vocab_size, ctx):
                     step_num, mlm_metric, nsp_metric, None)
                 begin_time = time.time()
                 running_mlm_loss = running_nsp_loss = running_num_tks = 0
-                mlm_metric.reset_running()
-                nsp_metric.reset_running()
+                mlm_metric.reset_local()
+                nsp_metric.reset_local()
 
     mx.nd.waitall()
     eval_end_time = time.time()


Test patch:
diff --git a/scripts/tests/test_bert.py b/scripts/tests/test_bert.py
--- a/scripts/tests/test_bert.py
+++ b/scripts/tests/test_bert.py
@@ -10,7 +10,7 @@
 def test_pretrain():
     # TODO(haibin) update test once MXNet 1.5 is released.
     try:
-        from mx.nd.contrib import mp_adamw_update
+        from mxnet.ndarray.contrib import mp_adamw_update
         process = subprocess.check_call(['python', './scripts/bert/create_pretraining_data.py',
                                          '--input_file', './scripts/bert/sample_text.txt',
                                          '--output_dir', 'test/bert/data',
Repository: dmlc/gluon-nlp

Issue:
Flaky Test:  test_datasets.py::test_conll2004
http://ci.mxnet.io/blue/organizations/jenkins/GluonNLP-py3-cpu-unittest/detail/PR-669/11/pipeline

```
        if overwrite or not os.path.exists(fname) or (sha1_hash and not check_sha1(fname, sha1_hash)):

            dirname = os.path.dirname(os.path.abspath(os.path.expanduser(fname)))

            if not os.path.exists(dirname):

                os.makedirs(dirname)

            while retries + 1 > 0:

                # Disable pyling too broad Exception

                # pylint: disable=W0703

                try:

                    print('Downloading {} from {}...'.format(fname, url))

                    r = requests.get(url, stream=True, verify=verify_ssl)

                    if r.status_code != 200:

>                       raise RuntimeError('Failed downloading url {}'.format(url))

E                       RuntimeError: Failed downloading url http://www.cs.upc.edu/~srlconll/st04/conll04st-release.tar.gz


```

Can we host it ourselves? I did not find license information for this file at http://www.lsi.upc.es/~srlconll/st04/st04.html 


Discussion:

Resolving patch:
diff --git a/src/gluonnlp/data/conll.py b/src/gluonnlp/data/conll.py
--- a/src/gluonnlp/data/conll.py
+++ b/src/gluonnlp/data/conll.py
@@ -31,7 +31,7 @@
 import tarfile
 
 from mxnet.gluon.data import SimpleDataset
-from mxnet.gluon.utils import download, check_sha1
+from mxnet.gluon.utils import download, check_sha1, _get_repo_file_url
 
 from .. import _constants as C
 from .registry import register
@@ -66,9 +66,12 @@ def _get_data(self):
         for data_file_name, data_hash in self._get_data_file_hash():
             root = self._root
             path = os.path.join(root, data_file_name)
+            if hasattr(self, 'namespace'):
+                url = _get_repo_file_url(self.namespace, archive_file_name)
+            else:
+                url = self.base_url + archive_file_name
             if not os.path.exists(path) or not check_sha1(path, data_hash):
-                download(self.base_url + archive_file_name,
-                         path=root, sha1_hash=archive_hash)
+                download(url, path=root, sha1_hash=archive_hash)
                 self._extract_archive()
             paths.append(path)
         return paths
@@ -355,6 +358,7 @@ def __init__(self, segment='train',
         super(CoNLL2004, self).__init__(segment, root)
 
     base_url = 'http://www.cs.upc.edu/~srlconll/st04/'
+    namespace = 'gluon/dataset/conll'
     codec = 'utf-8'
 
     def _get_data_file_hash(self):


Test patch:
Repository: dmlc/gluon-nlp

Issue:
[usability] print warnings when Pad() is used
When `data.batchify.Pad()` is used, the default padding value is 0. However, this usually does not respond to the pad token id in the vocab. We should print a warning if 0 is used for `Pad()` so that users are aware. 


Discussion:

Resolving patch:
diff --git a/src/gluonnlp/data/batchify/batchify.py b/src/gluonnlp/data/batchify/batchify.py
--- a/src/gluonnlp/data/batchify/batchify.py
+++ b/src/gluonnlp/data/batchify/batchify.py
@@ -20,7 +20,7 @@
 into batches for fast processing."""
 __all__ = ['Stack', 'Pad', 'Tuple']
 
-import logging
+import warnings
 
 import numpy as np
 import mxnet as mx
@@ -223,6 +223,12 @@ def __init__(self, axis=0, pad_val=0, ret_length=False, dtype=None):
         self._dtype = dtype
         self._warned = False
 
+        if pad_val == 0:
+            warnings.warn(
+                'Padding value 0 is used in data.batchify.Pad(). '
+                'Please check whether this is intended '
+                '(e.g. value of padding index in the vocabulary).')
+
     def __call__(self, data):
         """Batchify the input data.
 
@@ -251,7 +257,7 @@ def __call__(self, data):
 
         if isinstance(data[0], mx.nd.NDArray) and not self._warned:
             self._warned = True
-            logging.warning(
+            warnings.warn(
                 'Using Pad with NDArrays is discouraged for speed reasons. '
                 'Instead you should pad your data while it is still a list '
                 'and before converting to an NDArray. '


Test patch:
Repository: dmlc/gluon-nlp

Issue:
ATIS/SNIPS datasets and GLUE datasets don't appear in the website API doc 
http://gluon-nlp.mxnet.io/api/modules/data.html

does not show the details of ATISDataset/SNIPSDataset and GlueCoLA, GlueSST2, GlueSTSB, GlueQQP, GlueRTE, GlueMNLI, GlueQNLI, GlueWNLI



Discussion:

Resolving patch:
diff --git a/src/gluonnlp/data/__init__.py b/src/gluonnlp/data/__init__.py
--- a/src/gluonnlp/data/__init__.py
+++ b/src/gluonnlp/data/__init__.py
@@ -23,7 +23,7 @@
 from . import (batchify, candidate_sampler, conll, corpora, dataloader,
                dataset, question_answering, registry, sampler, sentiment,
                stream, transforms, translation, utils,
-               word_embedding_evaluation, intent_slot)
+               word_embedding_evaluation, intent_slot, glue)
 from .candidate_sampler import *
 from .conll import *
 from .glue import *
@@ -42,7 +42,8 @@
 from .intent_slot import *
 
 __all__ = (['batchify'] + utils.__all__ + transforms.__all__ + sampler.__all__
-           + dataset.__all__ + corpora.__all__ + sentiment.__all__ +
-           word_embedding_evaluation.__all__ + stream.__all__ + conll.__all__ +
-           translation.__all__ + registry.__all__ + question_answering.__all__
-           + dataloader.__all__ + candidate_sampler.__all__)
+           + dataset.__all__ + corpora.__all__ + sentiment.__all__
+           + word_embedding_evaluation.__all__ + stream.__all__ + conll.__all__
+           + translation.__all__ + registry.__all__ + question_answering.__all__
+           + dataloader.__all__ + candidate_sampler.__all__ + intent_slot.__all__
+           + glue.__all__)


Test patch:
Repository: dmlc/gluon-nlp

Issue:
Citation for converted BERT model
We shall explicitly cite the converted BERT models on the website to avoid confusions. For example, there are multiple ClinicalBERT model repositories and papers: 
https://github.com/EmilyAlsentzer/clinicalBERT
https://github.com/kexinhuang12345/clinicalBERT
@leezu 


Discussion:

Resolving patch:
diff --git a/src/gluonnlp/model/bert.py b/src/gluonnlp/model/bert.py
--- a/src/gluonnlp/model/bert.py
+++ b/src/gluonnlp/model/bert.py
@@ -747,7 +747,7 @@ def bert_12_768_12(dataset_name=None, vocab=None, pretrained=True, ctx=mx.cpu(),
         'openwebtext_book_corpus_wiki_en_uncased',
         'wiki_multilingual_uncased', 'wiki_multilingual_cased',
         'scibert_scivocab_uncased', 'scibert_scivocab_cased',
-        'scibert_basevocab_uncased','scibert_basevocab_cased',
+        'scibert_basevocab_uncased', 'scibert_basevocab_cased',
         'biobert_v1.0_pmc', 'biobert_v1.0_pubmed', 'biobert_v1.0_pubmed_pmc',
         'biobert_v1.1_pubmed',
         'clinicalbert'
@@ -789,6 +789,30 @@ def bert_12_768_12(dataset_name=None, vocab=None, pretrained=True, ctx=mx.cpu(),
         parameters will be left uninitialized. Otherwise AssertionError is
         raised.
 
+    The pretrained parameters for dataset_name
+    'openwebtext_book_corpus_wiki_en_uncased' were obtained by running the
+    GluonNLP BERT pre-training script on OpenWebText.
+
+    The pretrained parameters for dataset_name 'scibert_scivocab_uncased',
+    'scibert_scivocab_cased', 'scibert_basevocab_uncased',
+    'scibert_basevocab_cased' were obtained by converting the parameters
+    published by "Beltagy, I., Cohan, A., & Lo, K. (2019). Scibert: Pretrained
+    contextualized embeddings for scientific text. arXiv preprint
+    arXiv:1903.10676."
+
+    The pretrained parameters for dataset_name 'biobert_v1.0_pmc',
+    'biobert_v1.0_pubmed', 'biobert_v1.0_pubmed_pmc', 'biobert_v1.1_pubmed'
+    were obtained by converting the parameters published by "Lee, J., Yoon, W.,
+    Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2019). Biobert:
+    pre-trained biomedical language representation model for biomedical text
+    mining. arXiv preprint arXiv:1901.08746."
+
+    The pretrained parameters for dataset_name 'clinicalbert' were obtained by
+    converting the parameters published by "Huang, K., Altosaar, J., &
+    Ranganath, R. (2019). ClinicalBERT: Modeling Clinical Notes and Predicting
+    Hospital Readmission. arXiv preprint arXiv:1904.05342."
+
+
     Returns
     -------
     BERTModel, gluonnlp.vocab.BERTVocab


Test patch:
Repository: dmlc/gluon-nlp

Issue:
Unicode comma error
## Description
When running the `word_language_model.py` script, sometimes I ran into the following error:

```
Traceback (most recent call last):
  File "word_language_model.py", line 468, in <module>
    train()
  File "word_language_model.py", line 426, in train
    trainer.learning_rate))
UnicodeEncodeError: 'ascii' codec can't encode character '\uff0c' in position 62: ordinal not in range(128)
```
The error is caused by the following print statement:

```
print('[Epoch %d] time cost %.2fs, valid loss %.2f, valid ppl %.2f，lr %.2f' % (
            epoch, time.time() - start_epoch_time, val_L, math.exp(val_L),
            trainer.learning_rate))
```

After a careful inspection, we found the third comma is a chinese comma `，` in  `valid ppl %.2f，`.  To avoid potential running error, this comma needed to be replaced with an ASCII comma `,`.




Discussion:

Resolving patch:
diff --git a/scripts/language_model/word_language_model.py b/scripts/language_model/word_language_model.py
--- a/scripts/language_model/word_language_model.py
+++ b/scripts/language_model/word_language_model.py
@@ -421,7 +421,7 @@ def train():
         else:
             model.save_parameters('{}.val.params'.format(args.save))
         val_L = evaluate(val_data, val_batch_size, '{}.val.params'.format(args.save), context[0])
-        print('[Epoch %d] time cost %.2fs, valid loss %.2f, valid ppl %.2f，lr %.2f' % (
+        print('[Epoch %d] time cost %.2fs, valid loss %.2f, valid ppl %.2f, lr %.2f' % (
             epoch, time.time() - start_epoch_time, val_L, math.exp(val_L),
             trainer.learning_rate))
 


Test patch:
Repository: dmlc/gluon-nlp

Issue:
Deprecation warning due to invalid escape sequences
## Description

Deprecation warning due to invalid escape sequences. Using raw strings or escaping them again helps in resolving this. Check https://github.com/asottile/pyupgrade/ for automatic fix of this.

## To Reproduce

```
find . -iname '*.py' | grep -Ev 'rdf4|doc|benchmark|tool' | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./ci/batch/submit-job.py:95: DeprecationWarning: invalid escape sequence \-
  jobName = re.sub('[^A-Za-z0-9_\-]', '', args.name)[:128]  # Enforce AWS Batch jobName rules
```


Discussion:

Resolving patch:
diff --git a/ci/batch/submit-job.py b/ci/batch/submit-job.py
--- a/ci/batch/submit-job.py
+++ b/ci/batch/submit-job.py
@@ -92,7 +92,7 @@ def main():
     spin = ['-', '/', '|', '\\', '-', '/', '|', '\\']
     logGroupName = '/aws/batch/job'
 
-    jobName = re.sub('[^A-Za-z0-9_\-]', '', args.name)[:128]  # Enforce AWS Batch jobName rules
+    jobName = re.sub(r'[^A-Za-z0-9_\-]', '', args.name)[:128]  # Enforce AWS Batch jobName rules
     jobQueue = args.job_queue
     jobDefinition = args.job_definition
     command = args.command.split()


Test patch:
Repository: bornhack/bornhack-website

Issue:
Make event details available as part of .ics feed
![img_0061](https://user-images.githubusercontent.com/1938389/29740572-b002247a-8a59-11e7-9b8b-0615271cfbc5.PNG)

Make **event details** available as
1. a _description_ for .ics feed subscribers, **or**
2. a _direct link_ to the event details on the bornhack website in the .ics feed description.


Discussion:
An example from my Facebook feed .ics subscription 

![img_0062](https://user-images.githubusercontent.com/1938389/29740599-66448746-8a5a-11e7-90ec-0efd28140b5a.PNG)


Resolving patch:
diff --git a/src/program/models.py b/src/program/models.py
--- a/src/program/models.py
+++ b/src/program/models.py
@@ -518,6 +518,7 @@ def timeslots(self):
     def get_ics_event(self):
         ievent = icalendar.Event()
         ievent['summary'] = self.event.title
+        ievent['description'] = self.event.abstract
         ievent['dtstart'] = icalendar.vDatetime(self.when.lower).to_ical()
         ievent['dtend'] = icalendar.vDatetime(self.when.upper).to_ical()
         ievent['location'] = icalendar.vText(self.location.name)


Test patch:
Repository: bornhack/bornhack-website

Issue:
include iban/swift/bic info on custominvoice pdfs

include iban/swift/bic info on custominvoice pdfs



Discussion:

Resolving patch:
diff --git a/src/shop/invoiceworker.py b/src/shop/invoiceworker.py
--- a/src/shop/invoiceworker.py
+++ b/src/shop/invoiceworker.py
@@ -1,3 +1,4 @@
+from django.conf import settings
 from django.core.files import File
 from utils.pdf import generate_pdf_letter
 from shop.email import add_invoice_email, add_creditnote_email
@@ -39,6 +40,11 @@ def do_work():
                 template=template,
                 formatdict={
                     'invoice': invoice,
+                    'bank': settings.BANKACCOUNT_BANK,
+                    'bank_iban': settings.BANKACCOUNT_IBAN,
+                    'bank_bic': settings.BANKACCOUNT_SWIFTBIC,
+                    'bank_dk_reg': settings.BANKACCOUNT_REG,
+                    'bank_dk_accno': settings.BANKACCOUNT_ACCOUNT,
                 },
             )
             logger.info('Generated pdf for invoice %s' % invoice)


Test patch:
Repository: bornhack/bornhack-website

Issue:
backoffice: show quantity of products ordered after scanning qr code

backoffice: show quantity of products ordered after scanning qr code



Discussion:

Resolving patch:
diff --git a/src/tickets/models.py b/src/tickets/models.py
--- a/src/tickets/models.py
+++ b/src/tickets/models.py
@@ -93,8 +93,7 @@ def generate_pdf(self):
         formatdict = {"ticket": self}
 
         if self.ticket_type.single_ticket_per_product and self.shortname == "shop":
-            orp = self.get_orp()
-            formatdict["quantity"] = orp.quantity
+            formatdict["quantity"] = self.orp.quantity
 
         return generate_pdf_letter(
             filename="{}_ticket_{}.pdf".format(self.shortname, self.pk),
@@ -165,5 +164,6 @@ def get_absolute_url(self):
     def shortname(self):
         return "shop"
 
-    def get_orp(self):
+    @property
+    def orp(self):
         return OrderProductRelation.objects.get(product=self.product, order=self.order)


Test patch:
Repository: piskvorky/gensim

Issue:
HdpModel has no show_topics method and print_topics returns None
These methods should mirror the `LdaModel` `show_topics`and `print_topics`methods.

HdpModel has no show_topics method and print_topics returns None
These methods should mirror the `LdaModel` `show_topics`and `print_topics`methods.



Discussion:

Resolving patch:
diff --git a/gensim/models/hdpmodel.py b/gensim/models/hdpmodel.py
old mode 100644
new mode 100755
--- a/gensim/models/hdpmodel.py
+++ b/gensim/models/hdpmodel.py
@@ -459,11 +459,25 @@ def update_expectations(self):
         self.m_status_up_to_date = True
 
     def print_topics(self, topics=20, topn=20):
+        """Alias for `show_topics()` that prints the `topn` most
+        probable words for `topics` number of topics to log.
+        Set `topics=-1` to print all topics."""
+        return self.show_topics(topics=topics, topn=topn, log=True)
+        
+    def show_topics(self, topics=20, topn=20, log=False, formatted=True):
+        """
+        Print the `topN` most probable words for `topics` number of topics.
+        Set `topics=-1` to print all topics.
+
+        Set `formatted=True` to return the topics as a list of strings, or 
+        `False` as lists of (weight, word) pairs.
+
+        """        
         if not self.m_status_up_to_date:
             self.update_expectations()
         betas = self.m_lambda + self.m_eta
         hdp_formatter = HdpTopicFormatter(self.id2word, betas)
-        hdp_formatter.print_topics(topics, topn)
+        return hdp_formatter.show_topics(topics, topn, log, formatted)
 
     def save_topics(self, doc_count=None):
         """legacy method; use `self.save()` instead"""


Test patch: