internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pytorch/text

Issue:
Field.stop_words is overwritten and is no longer a set
https://github.com/pytorch/text/blob/758d356fd76ee6102de06a4cb5e414a3164c9e42/torchtext/data/field.py#L166-L173

I think this is an error and self.stop_words should not be overwritten after the condition.


Discussion:

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -163,14 +163,10 @@ def __init__(self, sequential=True, use_vocab=True, init_token=None,
         self.pad_token = pad_token if self.sequential else None
         self.pad_first = pad_first
         self.truncate_first = truncate_first
-        if stop_words is not None:
-            try:
-                self.stop_words = set(stop_words)
-            except TypeError:
-                raise ValueError("Stop words must be convertible to a set")
-        else:
-            self.stop_words = stop_words
-        self.stop_words = stop_words
+        try:
+            self.stop_words = set(stop_words) if stop_words is not None else None
+        except TypeError:
+            raise ValueError("Stop words must be convertible to a set")
         self.is_target = is_target
 
     def __getstate__(self):


Test patch:
Repository: pytorch/text

Issue:
Use "cpu" instead of None for representing device
I think using device value as `cpu` instead of `None` will make `Iterator` class compatible with PyTorch device types [1] i.e., "cpu" and "gpu". 

[1] https://pytorch.org/docs/stable/tensor_attributes.html 


Discussion:
Sure, could you submit a PR for this?

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -3,6 +3,7 @@
 
 import logging
 
+import torch
 from .utils import RandomShuffler
 from .batch import Batch
 from .dataset import Dataset
@@ -65,6 +66,12 @@ def __init__(self, dataset, batch_size, sort_key=None, device=None,
                            + " or passing a string as an argument. This behavior will be"
                            + " deprecated soon and currently defaults to cpu.")
             device = None
+
+        if device is None:
+            device = torch.device('cpu')
+        elif isinstance(device, str):
+            device = torch.device(device)
+
         self.device = device
         self.random_shuffler = RandomShuffler()
 


Test patch:
Repository: pytorch/text

Issue:
pip install torchtext==0.7.0 installs incompatible PyTorch 1.7.0
## 🐛 Bug
**Describe the bug**
Recently, after I do `pip install torchtext==0.7.0`, import torchtext would cause segmentation fault. I found that degrading pytorch to 1.6.0 fixes this issue. 

**To Reproduce**
Steps to reproduce the behavior:
1. `pip install torchtext==0.7.0` (assuming that pytorch is not installed yet, and this command will install the latest pytorch)
2. python -c "import torchtext"

**Expected behavior**
Segmentation Fault

**Environment**

 - PyTorch Version (e.g., 1.0): 1.7.0
 - OS (e.g., Linux): Linux/MacOS
 - Python: 3.8.3


Discussion:
Yup. That's expected. Since we add cpp extension in torchtext, we have to publish torchtext and pytorch together for compatibility. You should use torchtext=0.8.0 with torch=1.7.0. Any version before 0.7.0 has python only so the compatibility is not an issue. Feel free to re-open the issue if you still have questions.
But, shouldn't `pip install torchtext==0.7.0` install `pytorch==1.6` then?
> But, shouldn't `pip install torchtext==0.7.0` install `pytorch==1.6` then?

It should but obviously it didn't. If users would like the access to the previous version, please follow the version compatibility on README file of the repo. 
> > But, shouldn't `pip install torchtext==0.7.0` install `pytorch==1.6` then?
> 
> It should but obviously it didn't. If users would like the access to the previous version, please follow the version compatibility on README file of the repo.

Is it possible for you to fix this such that `pip install torchtext==0.7.0` automatically installs `pytorch==1.6` ? torchtext 0.7.0 was released only **a few months ago** and lots of users are still using that due to the overhaul of code in later versions (such as the retiring of `Field` and `Batch`).
Let me follow-up with @seemethere 
Yup looks like this is a real bug, we don't actually set a specific torch version in the setup.py

https://github.com/pytorch/text/blob/v0.8.0-rc2/setup.py#L85
This will be addressed in the next release to fix the torch/torchtext version.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -49,6 +49,12 @@ def _export_version(version, sha):
 
 print('-- Building version ' + VERSION)
 
+pytorch_package_version = os.getenv('PYTORCH_VERSION')
+
+pytorch_package_dep = 'torch'
+if pytorch_package_version is not None:
+    pytorch_package_dep += "==" + pytorch_package_version
+
 
 class clean(distutils.command.clean.clean):
     def run(self):
@@ -82,7 +88,7 @@ def run(self):
     license='BSD',
 
     install_requires=[
-        'tqdm', 'requests', 'torch', 'numpy'
+        'tqdm', 'requests', pytorch_package_dep, 'numpy'
     ],
     python_requires='>=3.5',
     classifiers=[


Test patch:
Repository: pytorch/text

Issue:
Make TorchText installable from direct reference
In order to make `pip install https://github.com/pytorch/text` succeed
`setup.py` should have logic to initialize submodules

Add `check_submodules()` routine to setup.py to check if
'third_party/re2/CMakeLists.txt' is present on the filesystem and
attempt to initialize submodules if it is not


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -4,6 +4,7 @@
 import os
 import shutil
 import subprocess
+import sys
 from pathlib import Path
 
 from build_tools import setup_helpers
@@ -44,6 +45,18 @@ def _export_version(version, sha):
         fileobj.write("git_version = {}\n".format(repr(sha)))
 
 
+def _init_submodule():
+    print(" --- Initializing submodules")
+    try:
+        subprocess.check_call(["git", "submodule", "init"])
+        subprocess.check_call(["git", "submodule", "update"])
+    except Exception:
+        print(" --- Submodule initalization failed")
+        print("Please run:\n\tgit submodule update --init --recursive")
+        sys.exit(1)
+    print(" --- Initialized submodule")
+
+
 VERSION, SHA = _get_version()
 _export_version(VERSION, SHA)
 
@@ -76,6 +89,7 @@ def run(self):
                 shutil.rmtree(str(path), ignore_errors=True)
 
 
+_init_submodule()
 setup_info = dict(
     # Metadata
     name="torchtext",


Test patch:
Repository: pytorch/text

Issue:
update documentation to reflect IMDB output
When attempting to use the IMDB api, I got results that were different from what the docs suggested. This PR attempts to update the docs with the correct output of the IMDB api.


Discussion:
@acxz I think this might have been an oversight when the dataset was implemented. To be consistent with our other text classification dataset, we do want to return labels as integers. 

If you want to take this on, it would be as simple as adding a  `MAP_LABELS` similar to how we do it in the QNLI dataset.
https://github.com/pytorch/text/blob/fd49d42c6c90fc4b556b63227f3509292bbdba11/torchtext/datasets/qnli.py#L40

And then we can do a simple label lookup in the `_modify_res` method like so
https://github.com/pytorch/text/blob/fd49d42c6c90fc4b556b63227f3509292bbdba11/torchtext/datasets/qnli.py#L59
Ah I see, I'll take a stab at this in the upcoming weekend. Thanks for the implementation pointers.
@acxz lmk if you still plan on taking this on, otherwise I'm happy to submit a PR for it this week 😄 
Yeah, actually if you could take it on that would be great. Ended up traveling this weekend and this week is looking a bit busy. Thanks for that!

Resolving patch:
diff --git a/torchtext/datasets/imdb.py b/torchtext/datasets/imdb.py
--- a/torchtext/datasets/imdb.py
+++ b/torchtext/datasets/imdb.py
@@ -20,6 +20,8 @@
     "test": 25000,
 }
 
+MAP_LABELS = {"neg": 1, "pos": 2}
+
 _PATH = "aclImdb_v1.tar.gz"
 
 DATASET_NAME = "IMDB"
@@ -50,7 +52,7 @@ def _cache_filepath_fn(root, decompressed_folder, split, x):
 
 
 def _modify_res(t):
-    return Path(t[0]).parts[-1], t[1]
+    return MAP_LABELS[Path(t[0]).parts[-1]], t[1]
 
 
 def filter_imdb_data(key, fname):


Test patch:
diff --git a/test/torchtext_unittest/datasets/test_imdb.py b/test/torchtext_unittest/datasets/test_imdb.py
--- a/test/torchtext_unittest/datasets/test_imdb.py
+++ b/test/torchtext_unittest/datasets/test_imdb.py
@@ -29,8 +29,8 @@ def _get_mock_dataset(root_dir):
         for i in range(5):
             # all negative labels are read first before positive labels in the
             # IMDB dataset implementation
-            label = "neg" if i < 2 else "pos"
-            cur_dir = pos_dir if label == "pos" else neg_dir
+            label = 1 if i < 2 else 2
+            cur_dir = pos_dir if label == 2 else neg_dir
             txt_file = os.path.join(cur_dir, f"{i}{i}_{i}.txt")
             with open(txt_file, "w", encoding="utf-8") as f:
                 rand_string = get_random_unicode(seed)
Repository: pytorch/text

Issue:
Implementing __contains__ for vocab.Vectors class
## 🚀 Feature

Isn't it better to implement \_\_contains\_\_ for Vectors class? In this way, one can easily find out whether a vocab is in the self.itos or not. 


Discussion:
Thanks for opening an issue @saeeddhqan! If you want to propose this feature, we can help with testing and approving. 

Resolving patch:
diff --git a/torchtext/vocab/vectors.py b/torchtext/vocab/vectors.py
--- a/torchtext/vocab/vectors.py
+++ b/torchtext/vocab/vectors.py
@@ -64,6 +64,9 @@ def __getitem__(self, token):
         else:
             return self.unk_init(torch.Tensor(self.dim))
 
+    def __contains__(self, token):
+        return token in self.stoi
+
     def cache(self, name, cache, url=None, max_vectors=None):
         import ssl
 


Test patch:
Repository: pytorch/text

Issue:
404 Client Error in IWSLT2017 and IWSLT2016
## 🐛 Bug

```
from torchtext.vocab import build_vocab_from_iterator

special_symbols = ['<unk>', '<pad>', '<bos>', '<eos>']

def yield_tokens(data_iter: Iterable) -> List[str]:
    for data_sample in data_iter:
        yield token_transform(data_sample)

train_iter = IWSLT2017(split="train")
build_vocab_from_iterator(yield_tokens(train_iter), min_freq=1, specials=special_symbols, special_first=True)
```

When I run above code, it shows below error:
`requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://drive.google.com/uc?id=12ycYSzLIG253AFN35Y6qoyf9wtkOjakp`

I think some download link for the data is broken.


Discussion:

Resolving patch:
diff --git a/torchtext/datasets/iwslt2017.py b/torchtext/datasets/iwslt2017.py
--- a/torchtext/datasets/iwslt2017.py
+++ b/torchtext/datasets/iwslt2017.py
@@ -11,7 +11,7 @@
     _wrap_split_argument,
 )
 
-URL = "https://drive.google.com/u/0/uc?id=12ycYSzLIG253AFN35Y6qoyf9wtkOjakp"
+URL = "https://fbk.sharepoint.com/sites/MTUnit/_layouts/15/download.aspx?SourceUrl=%2Fsites%2FMTUnit%2FShared%20Documents%2Fwebsites%2FWIT3%2Dlibrary%2F2017%2D01%2Dtrnmted%2Etgz"
 _PATH = "2017-01-trnmted.tgz"
 MD5 = "aca701032b1c4411afc4d9fa367796ba"
 


Test patch:
Repository: mozilla/telemetry-analysis-service

Issue:
Handle dropped database connections gracefully
In https://sentry.prod.mozaws.net/operations/atmo-stage/issues/383510/ we see that the database connection is closed prematurely and neither Django nor psycopg is able to recover from that.

The error is well known to be related to using process fork with open connections: http://pythonhosted.org/psycopg2/usage.html#thread-safety

This is related to https://github.com/ui/django-rq/issues/216 and https://github.com/celery/celery/blob/4d63867c8281e94c74dcdf84fe2a441eaed6671b/celery/fixups/django.py#L135-L192.


Discussion:

Resolving patch:
diff --git a/atmo/worker.py b/atmo/worker.py
new file mode 100644
--- /dev/null
+++ b/atmo/worker.py
@@ -0,0 +1,26 @@
+from django.db import connections, DatabaseError, InterfaceError
+from rq_retry import RetryWorker
+
+
+class ConnectionClosingRetryWorker(RetryWorker):
+    def close_database(self):
+        for connection in connections.all():
+            try:
+                connection.close()
+            except InterfaceError:
+                pass
+            except DatabaseError as exc:
+                str_exc = str(exc)
+                if 'closed' not in str_exc and 'not connected' not in str_exc:
+                    raise
+
+    def perform_job(self, *args, **kwargs):
+        self.close_database()
+        try:
+            return super().perform_job(*args, **kwargs)
+        finally:
+            self.close_database()
+
+    def work(self, *args, **kwargs):
+        self.close_database()
+        return super().work(*args, **kwargs)


Test patch:
Repository: mozilla/telemetry-analysis-service

Issue:
ATMO should pre-click my single SSH key
Would save me thousands of milliseconds every time I launch a cluster ;)


Discussion:

Resolving patch:
diff --git a/atmo/clusters/views.py b/atmo/clusters/views.py
--- a/atmo/clusters/views.py
+++ b/atmo/clusters/views.py
@@ -15,7 +15,13 @@
 
 @login_required
 def new_cluster(request):
-    if request.user.created_sshkeys.count() == 0:
+    initial = {
+        'identifier': '{}-telemetry-analysis'.format(user_display(request.user)),
+        'size': 1,
+    }
+    ssh_key_count = request.user.created_sshkeys.count()
+
+    if ssh_key_count == 0:
         messages.error(
             request,
             mark_safe(
@@ -25,10 +31,10 @@ def new_cluster(request):
             )
         )
         return redirect('keys-new')
-    initial = {
-        'identifier': '{}-telemetry-analysis'.format(user_display(request.user)),
-        'size': 1,
-    }
+    elif ssh_key_count == 1:
+        # If only 1 ssh key, make it pre-selected.
+        initial['ssh_key'] = request.user.created_sshkeys.values('pk')[0]['pk']
+
     form = NewClusterForm(
         request.user,
         initial=initial,


Test patch:
diff --git a/tests/test_clusters.py b/tests/test_clusters.py
--- a/tests/test_clusters.py
+++ b/tests/test_clusters.py
@@ -4,8 +4,10 @@
 from datetime import timedelta
 
 import pytest
-from django.utils import timezone
+from allauth.account.utils import user_display
+from django.contrib.messages import get_messages
 from django.core.urlresolvers import reverse
+from django.utils import timezone
 
 from atmo.clusters import models
 
@@ -32,6 +34,26 @@ def cluster_provisioner_mocks(mocker):
     }
 
 
+def test_cluster_form_defaults(client, test_user, ssh_key):
+    response = client.post(reverse('clusters-new'), {}, follow=True)
+
+    form = response.context['form']
+
+    assert form.errors
+    assert (form.initial['identifier'] ==
+            '{}-telemetry-analysis'.format(user_display(test_user)))
+    assert form.initial['size'] == 1
+    assert form.initial['ssh_key'] == ssh_key.id
+
+
+def test_no_keys_redirect(client, test_user):
+    response = client.post(reverse('clusters-new'), {}, follow=True)
+    assert response.status_code == 200
+    assert response.redirect_chain[-1] == (reverse('keys-new'), 302)
+    assert ('No SSH keys associated to you' in
+            [m for m in get_messages(response.wsgi_request)][0].message)
+
+
 def test_create_cluster(client, test_user, ssh_key, cluster_provisioner_mocks):
     start_date = timezone.now()
Repository: mozilla/telemetry-analysis-service

Issue:
Configure readbeat.scheduler logger to use mozlog
There is a logger in https://github.com/sibson/redbeat/blob/efc9491b2ffc6d9d544693143b42d771ec60f9c2/redbeat/schedulers.py#L61 that needs to be checked for using mozlog to show up in Kibana.


Discussion:

Resolving patch:
diff --git a/atmo/settings.py b/atmo/settings.py
--- a/atmo/settings.py
+++ b/atmo/settings.py
@@ -475,9 +475,14 @@ def LOGGING(self):
                     'handlers': ['console'],
                     'propagate': False,
                 },
-                'request.summary': {
+                'redbeat.schedulers': {
+                    'level': 'DEBUG',
                     'handlers': ['console'],
+                    'propagate': False,
+                },
+                'request.summary': {
                     'level': 'DEBUG',
+                    'handlers': ['console'],
                     'propagate': False,
                 },
             },


Test patch:
Repository: mozilla/telemetry-analysis-service

Issue:
ImportError: No module named 'atmo.clusters.jobs'
```
app@a898b116953a:~$ ./manage.py update_clusters
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 195, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 39, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 673, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/app/atmo/clusters/management/commands/update_clusters.py", line 6, in <module>
    from ...jobs import update_clusters
ImportError: No module named 'atmo.clusters.jobs'
```


Discussion:
Nice, just goes to show that nobody ever used this management command ;D

This is a simple fix, the clusters app was previously using RQ whose pattern is to use `jobs.py` files for the async jobs (not related to *Spark* jobs). The management command just needs a correct import:

```python
from ...tasks import update_clusters
# ...
```

Resolving patch:
diff --git a/atmo/clusters/management/commands/update_clusters.py b/atmo/clusters/management/commands/update_clusters.py
--- a/atmo/clusters/management/commands/update_clusters.py
+++ b/atmo/clusters/management/commands/update_clusters.py
@@ -3,7 +3,7 @@
 # file, you can obtain one at http://mozilla.org/MPL/2.0/.
 from django.core.management.base import BaseCommand
 
-from ...jobs import update_clusters
+from ...tasks import update_clusters
 
 
 class Command(BaseCommand):


Test patch:
Repository: mozilla/telemetry-analysis-service

Issue:
Use `select_for_update` in cluster expiration email task
There's a bit of a race condition [here](https://github.com/mozilla/telemetry-analysis-service/blob/e6fecbe12d09b2e2338ae62f5276b3b2f39b0b65/atmo/clusters/tasks.py#L38) where two tasks could ask for expiring clusters at the same time and start sending emails before they can be marked as sent. We should 1. wrap the whole task in a transaction and 2. use `select_for_update` on the query for active clusters to lock those rows.


Discussion:
Yay!

Resolving patch:
diff --git a/atmo/clusters/tasks.py b/atmo/clusters/tasks.py
--- a/atmo/clusters/tasks.py
+++ b/atmo/clusters/tasks.py
@@ -37,22 +37,23 @@ def deactivate_clusters():
 @celery.task
 def send_expiration_mails():
     deadline = timezone.now() + timedelta(hours=1)
-    soon_expired = Cluster.objects.active().filter(
-        end_date__lte=deadline,
-        expiration_mail_sent=False,
-    )
-    for cluster in soon_expired:
-        with transaction.atomic():
-            message = mail_builder.build_message(
-                'atmo/clusters/mails/expiration.mail', {
-                    'cluster': cluster,
-                    'deadline': deadline,
-                    'settings': settings,
-                },
-            )
-            message.send()
-            cluster.expiration_mail_sent = True
-            cluster.save()
+    with transaction.atomic():
+        soon_expired = Cluster.objects.select_for_update().active().filter(
+            end_date__lte=deadline,
+            expiration_mail_sent=False,
+        )
+        for cluster in soon_expired:
+            with transaction.atomic():
+                message = mail_builder.build_message(
+                    'atmo/clusters/mails/expiration.mail', {
+                        'cluster': cluster,
+                        'deadline': deadline,
+                        'settings': settings,
+                    },
+                )
+                message.send()
+                cluster.expiration_mail_sent = True
+                cluster.save()
 
 
 @celery.task(max_retries=3)


Test patch:
Repository: mozilla/telemetry-analysis-service

Issue:
Create view with jobs histories
There have been several jobs failing silently. Those jobs will soon generate alerts (#201) but it would  still be convenient to have a master view in the dashboard that shows the history, and their status, of all scheduled jobs.  Furthermore, every user should be able to see the history for their own jobs.


Discussion:
@vitillo Yep, another tab on the job detail page makes sense for the list of runs and their status. In fact the work on #201 is providing much of this and I'm far with the implementation so that we should finish that first and then extend that work to implement this feature request.

Would you consider showing the scheduled jobs of *all users* for admins in the dashboard useful? We have the permission system in place to enable the ability to do that.
> Would you consider showing the scheduled jobs of all users for admins in the dashboard useful? We have the permission system in place to enable the ability to do that.

Yes, in fact this is the view I was referring to in my comment. 
@vitillo Cool, I've opened #283 to track this separate issue then and leave this open for showing the history on the job detail page.

Resolving patch:
diff --git a/atmo/jobs/models.py b/atmo/jobs/models.py
--- a/atmo/jobs/models.py
+++ b/atmo/jobs/models.py
@@ -345,6 +345,7 @@ class SparkJobRun(EditedAtModel):
 
     class Meta:
         get_latest_by = 'created_at'
+        ordering = ['-created_at']
 
     __str__ = autostr('{self.jobflow_id}')
 


Test patch: