internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: tensorflow/tensor2tensor

Issue:
t2t-decoder bug when using decode_from_file
As said on gitter, there is a bug when decoding (I think it appeared in 1.2.5)
In the last batch it sent an empty line after the last line of the batch, and it throws this error:

INFO:tensorflow:Inference results INPUT: 
INFO:tensorflow:Inference results OUTPUT: Précisions
2017-10-24 21:19:07.312405: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: 
2017-10-24 21:19:07.312474: W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration: 
	 [[Node: PyFunc = PyFunc[Tin=[], Tout=[DT_INT32, DT_INT32], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"]()]]



Discussion:
@lukaszkaiser @rsepassi 
1.2.6 fixes the error BUT it does not fix the fact that decode_from_file with N lines output a decode_to_file with N+1 lines, because it will add an extra empty sequence at the end of the last batch.
Hope this is clear.
There are actually two different issues:
The first one was introduced by the custom delimiter commit.
here: https://github.com/tensorflow/tensor2tensor/commit/545ec342ed816fc4524dedd545380e0160a84720#diff-189703c3c1f690231282073cea1cec41R499

text.split(delimiter) will add an empty element at the end of the array.

easy fix is to modify the line after as:
inputs = [record.strip() for record in records[:-1]]
this solve the issue of the extra line as in my first post.
BUT
it does not solve the issue 
W tensorflow/core/framework/op_kernel.cc:1192] Out of range: exceptions.StopIteration:
which means there is another place where there is an index issue

help ....


Resolving patch:
diff --git a/tensor2tensor/utils/decoding.py b/tensor2tensor/utils/decoding.py
--- a/tensor2tensor/utils/decoding.py
+++ b/tensor2tensor/utils/decoding.py
@@ -512,7 +512,7 @@ def _get_sorted_inputs(filename, num_shards=1, delimiter="\n"):
   with tf.gfile.Open(decode_filename) as f:
     text = f.read()
     records = text.split(delimiter)
-    inputs = [record.strip() for record in records]
+    inputs = [record.strip() for record in records[:-1]]
   input_lens = [(i, len(line.split())) for i, line in enumerate(inputs)]
   sorted_input_lens = sorted(input_lens, key=operator.itemgetter(1))
   # We'll need the keys to rearrange the inputs back into their original order


Test patch:
Repository: tensorflow/tensor2tensor

Issue:
problem for compute the "real" BLEU 
it seems has some small problems.

Traceback (most recent call last):
File "/usr/local/bin/t2t-bleu", line 6, in
exec(compile(open(file).read(), file, 'exec'))
File "/workspace/tf_project/tensor2tensor/tensor2tensor/bin/t2t-bleu", line 137, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/workspace/tf_project/tensor2tensor/tensor2tensor/bin/t2t-bleu", line 80, in main
bleu = 100 * bleu_hook.bleu_wrapper(FLAGS.reference, FLAGS.translation, case_sensitive=False)
File "/workspace/tf_project/tensor2tensor/tensor2tensor/utils/bleu_hook.py", line 202, in bleu_wrapper
ref_tokens = [bleu_tokenize(x) for x in ref_lines]
File "/workspace/tf_project/tensor2tensor/tensor2tensor/utils/bleu_hook.py", line 187, in bleu_tokenize
string = UnicodeRegex.nondigit_punct_re.sub(r'\1 \2 ', string)
AttributeError: type object 'UnicodeRegex' has no attribute 'nondigit_punct_re'


Discussion:

Resolving patch:
diff --git a/tensor2tensor/utils/bleu_hook.py b/tensor2tensor/utils/bleu_hook.py
--- a/tensor2tensor/utils/bleu_hook.py
+++ b/tensor2tensor/utils/bleu_hook.py
@@ -153,7 +153,7 @@ def __init__(self):
     def _property_chars(prefix):
       return ''.join(six.unichr(x) for x in range(sys.maxunicode)
                      if unicodedata.category(six.unichr(x)).startswith(prefix))
-    punctuation = self._property_chars('P')
+    punctuation = _property_chars('P')
     self.nondigit_punct_re = re.compile(r'([^\d])([' + punctuation + r'])')
     self.punct_nondigit_re = re.compile(r'([' + punctuation + r'])([^\d])')
     self.symbol_re = re.compile('([' + _property_chars('S') + '])')
@@ -183,9 +183,10 @@ def bleu_tokenize(string):
   Returns:
     a list of tokens
   """
-  string = UnicodeRegex.nondigit_punct_re.sub(r'\1 \2 ', string)
-  string = UnicodeRegex.punct_nondigit_re.sub(r' \1 \2', string)
-  string = UnicodeRegex.symbol_re.sub(r' \1 ', string)
+  uregex = UnicodeRegex()
+  string = uregex.nondigit_punct_re.sub(r'\1 \2 ', string)
+  string = uregex.punct_nondigit_re.sub(r' \1 \2', string)
+  string = uregex.symbol_re.sub(r' \1 ', string)
   return string.split()
 
 


Test patch:
diff --git a/tensor2tensor/utils/bleu_hook_test.py b/tensor2tensor/utils/bleu_hook_test.py
--- a/tensor2tensor/utils/bleu_hook_test.py
+++ b/tensor2tensor/utils/bleu_hook_test.py
@@ -57,5 +57,9 @@ def testComputeMultipleNgrams(self):
     actual_bleu = 0.3436
     self.assertAllClose(bleu, actual_bleu, atol=1e-03)
 
+  def testBleuTokenize(self):
+    self.assertEqual(bleu_hook.bleu_tokenize(u'hi, “there”'), [u'hi', u',', u'“', u'there', u'”'])
+
+
 if __name__ == '__main__':
   tf.test.main()
Repository: tensorflow/tensor2tensor

Issue:
transformer_relative decoding fails: "Key transformer/transformer/... not found" 
I train `HPARAMS=transformer_relative_big` ([Self-Attention with Relative Position Representations](https://arxiv.org/abs/1803.02155)) and it goes OK,
but the decoding fails with
```
NotFoundError (see above for traceback): Key transformer/transformer/body/decoder/layer_0/encdec_attention/layer_prepostprocess/layer_norm/layer_norm_bias not found in checkpoint
         [[Node: save/RestoreV2_1 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:
0"](_arg_save/Const_0_0, save/RestoreV2_1/tensor_names, save/RestoreV2_1/shape_and_slices)]]
         [[Node: save/RestoreV2_1/_55 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnati
on=1, tensor_name="edge_62_save/RestoreV2_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
```
The checkpoint contains a key `transformer/body/decoder/layer_0/encdec_attention/layer_prepostprocess/layer_norm/layer_norm_bias`
but the decoding is looking for a key starting with **transformer/transformer**, i.e. there is one extra "transformer" name space.
The key is the first of decoder keys, so I guess it is not important that it is layer_norm_bias.

Interestingly, when I decode with HPARAMS=transformer_big_single_gpu (instead of the correct HPARAMS=transformer_relative_big), the decoding does not fail, it just produces very low BLEU, as could be expected because the model was not trained to get input absolute-position embeddings and at the same time it is missing the relative-position embeddings.

I made sure I am using the same T2T version and hparams for training and decoding.
tensor2tensor==1.5.6 (yes, it is not released yet, I am using it from #668)
tensorflow==1.6.0



Discussion:
Oh, fixing the bug was faster than writing this issue:-) (but I got the idea while writing the issue).
This issue will be automatically closed once #675 is merged.

Resolving patch:
diff --git a/tensor2tensor/models/transformer.py b/tensor2tensor/models/transformer.py
--- a/tensor2tensor/models/transformer.py
+++ b/tensor2tensor/models/transformer.py
@@ -225,6 +225,13 @@ def _beam_decode(self, features, decode_length, beam_size, top_beams, alpha):
               None if using greedy decoding (beam_size=1)
       }
     """
+    if self._hparams.self_attention_type != "dot_product":
+      # Caching is not guaranteed to work with attention types other than
+      # dot_product.
+      # TODO(petershaw): Support fast decoding when using relative
+      # position representations, i.e. "dot_product_relative" attention.
+      return self._beam_decode_slow(features, decode_length, beam_size,
+                                    top_beams, alpha)
     with tf.variable_scope(self.name):
       return self._fast_decode(
           features, decode_length, beam_size, top_beams, alpha)


Test patch:
Repository: tensorflow/tensor2tensor

Issue:
Python 3 compatibility issue with `self_generated_targets`
### Description

I get `AssertionError` when trying to return `self_generated_targets` in a model.

### Environment information

```
OS: macOS 10.13.4

$ pip freeze | grep tensor
mesh-tensorflow==0.0.4
tensor2tensor==1.11.0
tensorboard==1.12.0
tensorflow==1.12.0
tensorflow-metadata==0.9.0
tensorflow-probability==0.5.0

$ python -V
Python 3.6.4
```

### For bugs: error logs

```
  File "/Users/ywkim/tensor2tensor/tensor2tensor/utils/t2t_model.py", line 1266, in wrapping_model_fn                                                                 
    use_tpu=use_tpu)
  File "/Users/ywkim/tensor2tensor/tensor2tensor/utils/t2t_model.py", line 1340, in estimator_model_fn                                                                
    "Found {}".format(logits.keys())
AssertionError: Expect only key 'logits' when there is 'self_generated_targets'. Found dict_keys(['logits'])
```



Discussion:

Resolving patch:
diff --git a/tensor2tensor/utils/t2t_model.py b/tensor2tensor/utils/t2t_model.py
--- a/tensor2tensor/utils/t2t_model.py
+++ b/tensor2tensor/utils/t2t_model.py
@@ -1337,7 +1337,7 @@ def estimator_model_fn(cls,
       # by logits["self_generated_targets"].
       tf.logging.info("Replacing targets with model-provided targets.")
       features["targets"] = labels = logits.pop("self_generated_targets")
-      assert logits.keys() == ["logits"], (
+      assert list(logits.keys()) == ["logits"], (
           # See "Returns" in the "top" method docstring for the expected
           # "logits" format when targets are generated at training time.
           "Expect only key 'logits' when there is 'self_generated_targets'. "


Test patch:
Repository: tensorflow/tensor2tensor

Issue:
Undefined name 'loss' in the creation of 'training_loss_dict'
### Description

Undefined name __loss__ caused by an invalid comprehension in the code:
```python
training_loss_dict = average_sharded_losses([{
              "training": l
} for l in loss for loss in sharded_losses.values()])
```
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py#L341-L343

The above code is an invalid comprehension and will fail with a __NameError__ just like
```
python -c '[{"training": l} for l in loss for loss in "this is a test".split()]'
```

### Environment information

```
OS: https://github.com/tensorflow/tensor2tensor

$ python -V
# 3.7.0
```

### For bugs: reproduction and error logs

# Steps to reproduce:
$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__

# Error logs:
[flake8](http://flake8.pycqa.org) testing of https://github.com/tensorflow/tensor2tensor

$ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics__
```
./tensor2tensor/utils/t2t_model.py:227:22: F821 undefined name 'loss'
          } for l in loss for loss in sharded_losses.values()])
                     ^
1    F821 undefined name 'loss'
1
```


Discussion:

Resolving patch:
diff --git a/tensor2tensor/utils/t2t_model.py b/tensor2tensor/utils/t2t_model.py
--- a/tensor2tensor/utils/t2t_model.py
+++ b/tensor2tensor/utils/t2t_model.py
@@ -338,9 +338,9 @@ def model_fn_sharded(self, sharded_features):
             sharded_logits[k] = dp(self.top, v, datashard_to_features)
             sharded_losses[k] = dp(self.loss, sharded_logits[k],
                                    datashard_to_features)
-          training_loss_dict = average_sharded_losses([{
+          training_loss_dict = average_sharded_losses([({
               "training": l
-          } for l in loss for loss in sharded_losses.values()])
+          } for l in loss) for loss in sharded_losses.values()])
           losses.update(training_loss_dict)
         else:
           sharded_logits = dp(self.top, body_out, datashard_to_features)


Test patch:
Repository: mozilla/kitsune

Issue:
Move Product Details to the Database backend



Discussion:

Resolving patch:
diff --git a/kitsune/settings.py b/kitsune/settings.py
--- a/kitsune/settings.py
+++ b/kitsune/settings.py
@@ -60,6 +60,15 @@ def path(*parts):
 # Cache Settings
 CACHES = {
     'default': config('CACHE_URL', default='locmem://', cast=django_cache_url.parse),
+    'product-details': {
+        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
+        'LOCATION': 'product-details',
+        'OPTIONS': {
+            'MAX_ENTRIES': 200,  # currently 104 json files
+            'CULL_FREQUENCY':  4,  # 1/4 entries deleted if max reached
+        }
+
+    },
 }
 
 # Setting this to the Waffle version.
@@ -1061,3 +1070,7 @@ def read_only_mode(env):
 DMS_FIX_CURRENT_REVISIONS = config('DMS_FIX_CURRENT_REVISIONS', default=None)
 DMS_COHORT_ANALYSIS = config('DMS_COHORT_ANALYSIS', default=None)
 DMS_UPDATE_L10N_CONTRIBUTOR_METRICS = config('DMS_UPDATE_L10N_CONTRIBUTOR_METRICS', default=None)
+
+PROD_DETAILS_CACHE_NAME = 'product-details'
+PROD_DETAILS_STORAGE = config('PROD_DETAILS_STORAGE',
+                              default='product_details.storage.PDDatabaseStorage')


Test patch:
Repository: mozilla/kitsune

Issue:
Add gunicorn + Meinheld worker



Discussion:

Resolving patch:
diff --git a/wsgi/__init__.py b/wsgi/__init__.py
new file mode 100644
--- /dev/null
+++ b/wsgi/__init__.py
@@ -0,0 +1 @@
+from app import application  # noqa
diff --git a/wsgi/app.py b/wsgi/app.py
new file mode 100644
--- /dev/null
+++ b/wsgi/app.py
@@ -0,0 +1,29 @@
+"""
+WSGI config for kitsune project.
+
+It exposes the WSGI callable as a module-level variable named ``application``.
+
+For more information on this file, see
+https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/
+"""
+import os
+os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'kitsune.settings')  # NOQA
+
+from django.core.wsgi import get_wsgi_application
+
+import newrelic.agent
+from decouple import config
+from raven.contrib.django.raven_compat.middleware.wsgi import Sentry
+
+# For django-celery
+os.environ['CELERY_LOADER'] = 'django'
+
+application = get_wsgi_application()
+application = Sentry(application)
+
+# Add NewRelic
+newrelic_ini = config('NEW_RELIC_CONFIG_FILE', default='newrelic.ini')
+newrelic_license_key = config('NEW_RELIC_LICENSE_KEY', default=None)
+if newrelic_ini and newrelic_license_key:
+    newrelic.agent.initialize(newrelic_ini)
+    application = newrelic.agent.wsgi_application()(application)
diff --git a/wsgi/config.py b/wsgi/config.py
new file mode 100644
--- /dev/null
+++ b/wsgi/config.py
@@ -0,0 +1,11 @@
+# see http://docs.gunicorn.org/en/latest/configure.html#configuration-file
+
+from os import getenv
+
+
+bind = '0.0.0.0:{}'.format(getenv('PORT', 8000))
+workers = int(getenv('WSGI_NUM_WORKERS', 2))
+errorlog = '-'
+loglevel = getenv('WSGI_LOG_LEVEL', 'info')
+keepalive = int(getenv('WSGI_KEEP_ALIVE', 2))
+worker_class = getenv('GUNICORN_WORKER_CLASS', 'meinheld.gmeinheld.MeinheldWorker')


Test patch:
Repository: mozilla/kitsune

Issue:
Add whitenoise and serve static files from the app
Add whitenoise to serve static files in the dev (and stage?) instances. 

In production we'll probably go with another more efficient approach which is being investigated in #2949.

Whitenoise activation should be configurable via the `ENABLE_WHITENOISE` env variable.


Discussion:

Resolving patch:
diff --git a/wsgi/app.py b/wsgi/app.py
--- a/wsgi/app.py
+++ b/wsgi/app.py
@@ -21,6 +21,10 @@
 application = get_wsgi_application()
 application = Sentry(application)
 
+if config('ENABLE_WHITENOISE', default=False, cast=bool):
+    from whitenoise.django import DjangoWhiteNoise
+    application = DjangoWhiteNoise(application)
+
 # Add NewRelic
 newrelic_ini = config('NEW_RELIC_CONFIG_FILE', default='newrelic.ini')
 newrelic_license_key = config('NEW_RELIC_LICENSE_KEY', default=None)


Test patch:
Repository: mozilla/kitsune

Issue:
Redirect support.mozilla.com to support.mozilla.org



Discussion:
https://bugzilla.mozilla.org/show_bug.cgi?id=1428134
How do we plan to make this redirect? @metadave can you do this on the k8s level?
We talked a bit more in the meeting and decided to do it via a middleware in kitsune itself. This has a few benefits:

* Nothing else to run and monitor
* No need to make sure one is running in any other region
* It's fairly trivial to write as a Django Middleware

I'll see if I can knock it out soon.
I believe we can just use [this app](https://github.com/dabapps/django-enforce-host). I have a middleware I've used in another project, but this is basically that and just installable. 

Resolving patch:
diff --git a/kitsune/settings.py b/kitsune/settings.py
--- a/kitsune/settings.py
+++ b/kitsune/settings.py
@@ -475,6 +475,7 @@ def path(*parts):
     'multidb.middleware.PinningRouterMiddleware',
     'django_statsd.middleware.GraphiteMiddleware',
     'commonware.request.middleware.SetRemoteAddrFromForwardedFor',
+    'enforce_host.EnforceHostMiddleware',
 
     # LocaleURLMiddleware requires access to request.user. These two must be
     # loaded before the LocaleURLMiddleware
@@ -1037,6 +1038,12 @@ def JINJA_CONFIG():
 ]
 
 ALLOWED_HOSTS = config('ALLOWED_HOSTS', default='', cast=Csv())
+# in production set this to 'support.mozilla.org' and all other domains will redirect.
+# can be a comma separated list of allowed domains.
+# the first in the list will be the target of redirects.
+# needs to be None if not set so that the middleware will
+# be turned off. can't set default to None because of the Csv() cast.
+ENFORCE_HOST = config('ENFORCE_HOST', default='', cast=Csv()) or None
 
 # Allows you to specify waffle settings in the querystring.
 WAFFLE_OVERRIDE = config('WAFFLE_OVERRIDE', default=DEBUG, cast=bool)


Test patch:
Repository: mozilla/kitsune

Issue:
Improve performance of _get_creator_counts util function
`kitsune.community.utils._get_creator_counts` until function is DB heavy and takes a lot of time to execute. Evaluate its usefulness and provide a way to optimize the query and/or cache the results.  

This issue is related to the degraded performance SUMO experienced on Fri March 30th ([NR Error](https://rpm.newrelic.com/accounts/1299394/applications/45097089/downtime/34422892))


Discussion:
There was another degraded performance issue just now related to DB slowness that is related to this.

Resolving patch:
diff --git a/kitsune/community/utils.py b/kitsune/community/utils.py
--- a/kitsune/community/utils.py
+++ b/kitsune/community/utils.py
@@ -1,6 +1,8 @@
 import hashlib
 
 from datetime import datetime, date, timedelta
+from operator import itemgetter
+
 from django.conf import settings
 from django.core.cache import cache
 from django.db.models import Count, F
@@ -141,4 +143,8 @@ def _get_creator_counts(query, count, page):
 
         results.append(data)
 
+    # Descending Order the list according to count.
+    # As the top number of contributor should be at first
+    results = sorted(results, key=itemgetter('count'), reverse=True)
+
     return results, total


Test patch:
Repository: mozilla/kitsune

Issue:
Remove all references to  ethn.io and Treejack
Confirm is ethn.io is something we still use, remove otherwise

https://github.com/mozilla/kitsune/blob/master/kitsune/products/jinja2/products/product.html#L92

- [ ] remove treejack waffle.switch
- [ ] remove waffle.flag('ethnio-all')
Remove all references to  ethn.io and Treejack
Confirm is ethn.io is something we still use, remove otherwise

https://github.com/mozilla/kitsune/blob/master/kitsune/products/jinja2/products/product.html#L92

- [ ] remove treejack waffle.switch
- [ ] remove waffle.flag('ethnio-all')


Discussion:
Per Joni via email "We no longer need ethnio or Treejack. They were from past experiments."
Thanks Ben!
Per Joni via email "We no longer need ethnio or Treejack. They were from past experiments."
Thanks Ben!

Resolving patch:
diff --git a/kitsune/sumo/migrations/0002_initial_data.py b/kitsune/sumo/migrations/0002_initial_data.py
--- a/kitsune/sumo/migrations/0002_initial_data.py
+++ b/kitsune/sumo/migrations/0002_initial_data.py
@@ -23,19 +23,6 @@ def remove_ratelimit_bypass_perm(apps, schema_editor):
     perm = Permission.objects.filter(codename='bypass_ratelimit').delete()
 
 
-def create_treejack_switch(apps, schema_editor):
-    Switch = apps.get_model('waffle', 'Switch')
-    Switch.objects.create(
-        name='treejack',
-        note='Enables/disables the Treejack snippet.',
-        active=False)
-
-
-def remove_treejack_switch(apps, schema_editor):
-    Switch = apps.get_model('waffle', 'Switch')
-    Switch.objects.filter(name='treejack').delete()
-
-
 def create_refresh_survey_flag(apps, schema_editor):
     Sample = apps.get_model('waffle', 'Sample')
     Sample.objects.get_or_create(
@@ -60,6 +47,5 @@ class Migration(migrations.Migration):
 
     operations = [
         migrations.RunPython(create_ratelimit_bypass_perm, remove_ratelimit_bypass_perm),
-        migrations.RunPython(create_treejack_switch, remove_treejack_switch),
         migrations.RunPython(create_refresh_survey_flag, remove_refresh_survey_flag),
     ]


Test patch: