internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getsentry/snuba

Issue:
Snuba cleanup for sentry onpremise
### Environment

Sentry self-hosted 21.3.0 (based on docker-compose from here https://github.com/getsentry/onpremise/blob/21.3.0/docker-compose.yml)

### Steps to Reproduce

1) Setup all containers and up snuba-cleanup container
2) Check logs for snuba-cleanup: Every 5 minutes in log - `Dropped 0 partitions on None`
It looks like variable CLICKHOUSE_HOST is ignored here
https://github.com/getsentry/snuba/blob/41d7fe76aaf8a594e8f6e84015607dcde3f67ad4/snuba/cli/cleanup.py#L13
After manual run command in container - `snuba cleanup --clickhouse-host CLICKHOUSE_HOST_HERE --dry-run True`
i got `Dropped 0 partitions on CLICKHOUSE_HOST_HERE`

### Expected Result

Pass variable https://github.com/getsentry/onpremise/blob/bdd2686021cfea07507bc07d2756ac34a775c680/docker-compose.yml#L44 into cleanup command

### Actual Result

variable is `None` instead of clickhouse host

I'am not sure, bug this or not.


Discussion:
This just seems like a bug on the log statement here: https://github.com/getsentry/snuba/blob/41d7fe76aaf8a594e8f6e84015607dcde3f67ad4/snuba/cli/cleanup.py#L71

Which assumes `clickhouse_host` is always set unless an exception was raised above. I'll see if I can get a quick fix.

Resolving patch:
diff --git a/snuba/cli/cleanup.py b/snuba/cli/cleanup.py
--- a/snuba/cli/cleanup.py
+++ b/snuba/cli/cleanup.py
@@ -50,7 +50,8 @@ def cleanup(
 
     (clickhouse_user, clickhouse_password,) = storage.get_cluster().get_credentials()
 
-    database = storage.get_cluster().get_database()
+    cluster = storage.get_cluster()
+    database = cluster.get_database()
 
     if clickhouse_host and clickhouse_port:
         connection = ClickhousePool(
@@ -60,12 +61,12 @@ def cleanup(
             clickhouse_password,
             database,
         )
-    elif not storage.get_cluster().is_single_node():
+    elif not cluster.is_single_node():
         raise click.ClickException("Provide ClickHouse host and port for cleanup")
     else:
-        connection = storage.get_cluster().get_query_connection(
+        connection = cluster.get_query_connection(
             ClickhouseClientSettings.CLEANUP
         )
 
     num_dropped = run_cleanup(connection, storage, database, dry_run=dry_run)
-    logger.info("Dropped %s partitions on %s" % (num_dropped, clickhouse_host))
+    logger.info("Dropped %s partitions on %s" % (num_dropped, cluster))


Test patch:
Repository: getsentry/snuba

Issue:
About the sessions-subscription-results subscription issue
### Environment

- sentry | snuba version ：23.1.1

https://github.com/getsentry/snuba/pull/2737 ，@lynnagara Hello, I have a question about this pr, hope to get your answer, thank you very much

- After removing subscriptions-scheduler-executor-session support in snuba, how to write data to the topic of sessions-subscription-results? Because I see that the crash rate warning code in sentry is still there and has not changed, for example

- https://github.com/getsentry/sentry/pull/28526

https://github.com/getsentry/sentry/blob/8e00dcdf463d916b9ca79ddbe13e99f161d58db1/src/sentry/snuba/query_subscription_consumer.py#L61-L61

My original question is as follows, I have enabled the organizations:incidents function in sentry and subscribed to sessions-results through the following script
```bash
sentry
run
query-subscription-consumer
--topic=sessions-subscription-results
```
Because there is no data in the sessions-subscription-results topic, the crash rate alarm cannot work
<img width="1568" alt="image" src="https://user-images.githubusercontent.com/18591662/216570393-64748a25-1cd4-4980-966c-f7665dc8482b.png">








Discussion:

Resolving patch:
diff --git a/snuba/cli/subscriptions_scheduler_executor.py b/snuba/cli/subscriptions_scheduler_executor.py
--- a/snuba/cli/subscriptions_scheduler_executor.py
+++ b/snuba/cli/subscriptions_scheduler_executor.py
@@ -24,7 +24,7 @@
     "--dataset",
     "dataset_name",
     required=True,
-    type=click.Choice(["events", "transactions", "metrics"]),
+    type=click.Choice(["events", "transactions", "metrics", "sessions"]),
     help="The dataset to target.",
 )
 @click.option(
@@ -32,7 +32,9 @@
     "entity_names",
     required=True,
     multiple=True,
-    type=click.Choice(["events", "transactions", "metrics_counters", "metrics_sets"]),
+    type=click.Choice(
+        ["events", "transactions", "metrics_counters", "metrics_sets", "sessions"]
+    ),
     help="The entity to target.",
 )
 @click.option(


Test patch:
Repository: getsentry/snuba

Issue:
Support Redis TLS in Snuba
Hi,

Sentry Relay got support for Redis TLS in release 23.1.1.

Sentry got support for Redis TLS in release 23.4.0 - or at least I think, but please correct me if I'm wrong.

When can we get Redis TLS support for Snuba as currently, I don't see a way to define this setting based on this block of code extracted from the `master` branch (link = https://github.com/getsentry/snuba/blob/master/snuba/redis.py#L84-L94)

```
_default_redis_client: RedisClientType = _initialize_redis_cluster(
    {
        "use_redis_cluster": settings.USE_REDIS_CLUSTER,
        "cluster_startup_nodes": settings.REDIS_CLUSTER_STARTUP_NODES,
        "host": settings.REDIS_HOST,
        "port": settings.REDIS_PORT,
        "password": settings.REDIS_PASSWORD,
        "db": settings.REDIS_DB,
        "reinitialize_steps": settings.REDIS_REINITIALIZE_STEPS,
    }
)
```

Thanks!


Discussion:

Resolving patch:
diff --git a/snuba/redis.py b/snuba/redis.py
--- a/snuba/redis.py
+++ b/snuba/redis.py
@@ -77,6 +77,7 @@ def _initialize_redis_cluster(config: settings.RedisClusterConfig) -> RedisClien
             port=config["port"],
             password=config["password"],
             db=config["db"],
+            ssl=config.get("ssl", False),
             socket_keepalive=True,
         )
 
@@ -89,6 +90,7 @@ def _initialize_redis_cluster(config: settings.RedisClusterConfig) -> RedisClien
         "port": settings.REDIS_PORT,
         "password": settings.REDIS_PASSWORD,
         "db": settings.REDIS_DB,
+        "ssl": settings.REDIS_SSL,
         "reinitialize_steps": settings.REDIS_REINITIALIZE_STEPS,
     }
 )
diff --git a/snuba/settings/__init__.py b/snuba/settings/__init__.py
--- a/snuba/settings/__init__.py
+++ b/snuba/settings/__init__.py
@@ -127,6 +127,7 @@ class RedisClusterConfig(TypedDict):
     port: int
     password: str | None
     db: int
+    ssl: bool
     reinitialize_steps: int
 
 
@@ -140,6 +141,7 @@ class RedisClusterConfig(TypedDict):
 REDIS_PORT = int(os.environ.get("REDIS_PORT", 6379))
 REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD")
 REDIS_DB = int(os.environ.get("REDIS_DB", 1))
+REDIS_SSL = bool(os.environ.get("REDIS_SSL", False))
 REDIS_INIT_MAX_RETRIES = 3
 REDIS_REINITIALIZE_STEPS = 10
 


Test patch:
Repository: pytorch/text

Issue:
Possible bug in LanguageModelingDataset
In the code for [`LanguageModelingDataset`](https://github.com/pytorch/text/blob/master/torchtext/datasets/language_modeling.py), the original text seems to be pre-processed twice, viz.:

- `text += text_field.preprocess(line)` [at line 22](https://github.com/pytorch/text/blob/master/torchtext/datasets/language_modeling.py#L22)
- `examples = [data.Example.fromlist([text], fields)]` [at line 26](https://github.com/pytorch/text/blob/master/torchtext/datasets/language_modeling.py#L26), which in turn calls 
`setattr(ex, name, field.preprocess(val))` [at line 44 of example.py](https://github.com/pytorch/text/blob/master/torchtext/data/example.py#L44)

In fact, if I try to create a simple LanguageModelingDataset, I am getting an error as follows:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/riddasgu/.local/lib/python2.7/site-packages/torchtext/datasets/language_modeling.py", line 28, in __init__
    examples = [data.Example.fromlist([text], fields)]
  File "/home/riddasgu/.local/lib/python2.7/site-packages/torchtext/data/example.py", line 44, in fromlist
    setattr(ex, name, field.preprocess(val))
  File "/home/riddasgu/.local/lib/python2.7/site-packages/torchtext/data/field.py", line 91, in preprocess
    x = self.tokenize(x)
  File "/home/riddasgu/.local/lib/python2.7/site-packages/torchtext/data/field.py", line 63, in <lambda>
    tokenize=(lambda s: s.split()), include_lengths=False,
AttributeError: 'list' object has no attribute 'split'
```


Discussion:
Looks like that was my mistake in a recent patch. What I think is supposed to happen here is that the first `preprocess` does the tokenization (it has to be this way because we need to split the corpus into segments with equal numbers of tokens) and the second one should be a no-op because [this line](https://github.com/pytorch/text/blob/master/torchtext/data/field.py#L90) used to include a check that the field data was a string. I'll add that back in.

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -87,7 +87,7 @@ def preprocess(self, x):
         `preprocessing` Pipeline."""
         if six.PY2 and isinstance(x, six.string_types):
             x = Pipeline(lambda s: unicode(s, encoding='utf-8'))(x)
-        if self.sequential:
+        if self.sequential and isinstance(x, six.text_type):
             x = self.tokenize(x)
         if self.lower:
             x = Pipeline(six.text_type.lower)(x)


Test patch:
Repository: pytorch/text

Issue:
Prevent decoding unicode strs to unicode in Py2
Fixes #67 


Discussion:

Resolving patch:
diff --git a/torchtext/data/field.py b/torchtext/data/field.py
--- a/torchtext/data/field.py
+++ b/torchtext/data/field.py
@@ -85,7 +85,8 @@ def preprocess(self, x):
         first. If `sequential=True`, it will be tokenized. Then the input
         will be optionally lowercased and passed to the user-provided
         `preprocessing` Pipeline."""
-        if six.PY2 and isinstance(x, six.string_types):
+        if (six.PY2 and isinstance(x, six.string_types) and not
+                isinstance(x, unicode)):
             x = Pipeline(lambda s: unicode(s, encoding='utf-8'))(x)
         if self.sequential and isinstance(x, six.text_type):
             x = self.tokenize(x)


Test patch:
diff --git a/test/data/test_field.py b/test/data/test_field.py
--- a/test/data/test_field.py
+++ b/test/data/test_field.py
@@ -24,6 +24,11 @@ def test_preprocess(self):
                                           preprocessing=preprocess_pipeline)
         assert field_not_sequential.preprocess("Test string.") == "test string.!"
 
+        # Non-regression test that we do not try to decode unicode strings to unicode
+        field_not_sequential = data.Field(sequential=False, lower=True,
+                                          preprocessing=preprocess_pipeline)
+        assert field_not_sequential.preprocess(u"Test string.") == "test string.!"
+
     def test_pad(self):
         # Default case.
         field = data.Field()
Repository: pytorch/text

Issue:
Consistency with sorting: `sort=True`
**Problem:**
```
    train_iter, dev_iter, test_iter = data.BucketIterator.splits(
        (train, dev, test),
        batch_sizes=(32, 256, 256),
        sort_key=lambda x: len(x.input),
        sort=True,
        device=-1)  # Use CPU
```

If `sort=True` and `train=True`, then the `train_iter` batches are shuffled. This behavior is unexpected.

**Cause:**
Because by default `self.shuffle=True` is `train=True`. Then https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py#L113 `shuffle` overrides `sort`.

**Possible Solution:**
`sort=True` should override `shuffle=None and train=True`.


Discussion:
I think you're right, but I'll look into it more on Monday (a couple times a seemingly obvious, one-line change like this has broken existing code and I want to make sure that's unlikely here).

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -110,10 +110,10 @@ def splits(cls, datasets, batch_sizes=None, **kwargs):
 
     def data(self):
         """Return the examples in the dataset in order, sorted, or shuffled."""
-        if self.shuffle:
-            xs = [self.dataset[i] for i in self.random_shuffler(range(len(self.dataset)))]
-        elif self.sort:
+        if self.sort:
             xs = sorted(self.dataset, key=self.sort_key)
+        elif self.shuffle:
+            xs = [self.dataset[i] for i in self.random_shuffler(range(len(self.dataset)))]
         else:
             xs = self.dataset
         return xs


Test patch:
Repository: pytorch/text

Issue:
min_freq=0 bug
**Noticed:**
```
>>>some_field.build_vocab(some_dataset, min_freq=0)
>>>padding_idx = some_field.vocab.stoi['<pad'>]
>>>print(padding_idx, '<pad>')
12 <pad>
```

Looks like <pad> is not equal to 1 which is not okay.

Printed `stoi` and `itos` as well:
```
>>>print(some_field.vocab.stoi)
defaultdict(<function Vocab.__init__.<locals>.<lambda> at 0x103f4f0d0>, {'<pad>': 12, '1': 2, '2': 3, '9': 4, '0': 5, '5': 6, '4': 7, '6': 8, '8': 9, '3': 10, '7': 11, '<unk>': 13})
>>>print(some_field.vocab.itos)
['<unk>', '<pad>', '1', '2', '9', '0', '5', '4', '6', '8', '3', '7', '<pad>', '<unk>']
```

**Possible reason:**
Counter subtract does remove the specials but puts their count at 0.
`counter.subtract({tok: counter[tok] for tok in ['<unk>'] + specials})`

**Possible solution:**
Throw an error if `min_freq < 1`


Discussion:
Thanks! Will fix.

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -113,6 +113,7 @@ def __init__(self, counter, max_size=None, min_freq=1, wv_dir=os.getcwd(),
         """
         self.freqs = counter.copy()
         self.unk_init = unk_init
+        min_freq = max(min_freq, 1)
         counter.update(['<unk>'] + specials)
 
         if wv_type is not None:


Test patch:
Repository: pytorch/text

Issue:
Length of iterator fails in Python 2
The division `len(dataset) / batch_size` will be cast to int in python2, so that `math.ceil` doesn't really work when `len(dataset)` is not a multiple of batch size.


Discussion:
Yeah, this is a bug. The solution should be to wrap batch_size in `float`. Will fix.
I think it's preferable to just `from __future__ import division`?
Yes, thanks. I'm not very conversant in the Py2 world...

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -1,3 +1,5 @@
+from __future__ import division
+
 import math
 import random
 from contextlib import contextmanager


Test patch:
Repository: pytorch/text

Issue:
python2 iterator bug?
here is the snippet:

```
from __future__ import print_function
from torchtext.datasets import WikiText2

i_train, i_dev, i_test = WikiText2.iters(device='-1', root="data/")
for i in i_train:
    print(i)
```

here's the error:
```
Traceback (most recent call last):
  File "bug.py", line 5, in <module>
    for i in i_train:
  File "/u/bosctom/.local/lib/python2.7/site-packages/torchtext/data/iterator.py", line 218, in __iter__
    self.batch_size - len(text)))
TypeError: can't multiply sequence by non-int of type 'float'
```

The bug is related to python2's math.ceil() returning a float?


Discussion:
what version of torchtext are you using? A git hash would be nice, if you have one (or did you install from pip)? I get a different error related to the `WikiText2` dataset not being updated to use the latest Vocab API on Python 2 and 3.
I got that error using the pip version: 0.1.1
yes, you're right --- there needs to be an `int` call at https://github.com/pytorch/text/blob/0634892b91bbac24c15b2d75147f2c47b24fd0dd/torchtext/data/iterator.py#L220. Would you be willing to create a PR, or should i go ahead?

Resolving patch:
diff --git a/torchtext/data/iterator.py b/torchtext/data/iterator.py
--- a/torchtext/data/iterator.py
+++ b/torchtext/data/iterator.py
@@ -217,8 +217,8 @@ def __iter__(self):
         text = self.dataset[0].text
         TEXT = self.dataset.fields['text']
         TEXT.eos_token = None
-        text = text + ([TEXT.pad_token] * (math.ceil(len(text) / self.batch_size) *
-                                           self.batch_size - len(text)))
+        text = text + ([TEXT.pad_token] * int(math.ceil(len(text) / self.batch_size) *
+                                              self.batch_size - len(text)))
         data = TEXT.numericalize(
             [text], device=self.device, train=self.train)
         data = data.view(self.batch_size, -1).t().contiguous()


Test patch:
Repository: pytorch/text

Issue:
AttributeError: 'list' object has no attribute 'rstrip'
Hi all, previously torchtext works for me when I'm running anaconda python. However, now, when i uninstalled my anaconda python. It stops working.

It gives me the following error:  

```
File "/Library/Python/2.7/site-packages/torchtext/data/example.py", line 59, in fromlist
    setattr(ex, name, field.preprocess(val.rstrip('\n')))
AttributeError: 'list' object has no attribute 'rstrip'

```

Thanks!


Discussion:

Resolving patch:
diff --git a/torchtext/data/example.py b/torchtext/data/example.py
--- a/torchtext/data/example.py
+++ b/torchtext/data/example.py
@@ -56,7 +56,9 @@ def fromlist(cls, data, fields):
         ex = cls()
         for (name, field), val in zip(fields, data):
             if field is not None:
-                setattr(ex, name, field.preprocess(val.rstrip('\n')))
+                if isinstance(val, six.string_types):
+                    val = val.rstrip('\n')
+                setattr(ex, name, field.preprocess(val))
         return ex
 
     @classmethod


Test patch:
Repository: pytorch/text

Issue:
Escape csv header lines
I haven't been able to see how to skip first csv line in case of loading from a file with header. I could of course preprocess the file, but it'd be nice if there was an option to TabularDataset to tell it to skip the first line.


Discussion:
Oh, that's definitely worth including.

Resolving patch:
diff --git a/torchtext/data/dataset.py b/torchtext/data/dataset.py
--- a/torchtext/data/dataset.py
+++ b/torchtext/data/dataset.py
@@ -124,14 +124,14 @@ def download(cls, root, check=None):
 class TabularDataset(Dataset):
     """Defines a Dataset of columns stored in CSV, TSV, or JSON format."""
 
-    def __init__(self, path, format, fields, **kwargs):
+    def __init__(self, path, format, fields, skip_header=False, **kwargs):
         """Create a TabularDataset given a path, file format, and field list.
 
         Arguments:
             path (str): Path to the data file.
             format (str): The format of the data file. One of "CSV", "TSV", or
                 "JSON" (case-insensitive).
-            fields (list(tuple(str, Field)) or dict[str, (name, Field)]: For CSV and
+            fields (list(tuple(str, Field)) or dict[str: tuple(str, Field)]: For CSV and
                 TSV formats, list of tuples of (name, field). The list should be in
                 the same order as the columns in the CSV or TSV file, while tuples of
                 (name, None) represent columns that will be ignored. For JSON format,
@@ -139,12 +139,15 @@ def __init__(self, path, format, fields, **kwargs):
                 (name, field). This allows the user to rename columns from their JSON key
                 names and also enables selecting a subset of columns to load
                 (since JSON keys not present in the input dictionary are ignored).
+            skip_header (bool): Whether to skip the first line of the input file.
         """
         make_example = {
             'json': Example.fromJSON, 'dict': Example.fromdict,
             'tsv': Example.fromTSV, 'csv': Example.fromCSV}[format.lower()]
 
         with io.open(os.path.expanduser(path), encoding="utf8") as f:
+            if skip_header:
+                next(f)
             examples = [make_example(line, fields) for line in f]
 
         if make_example in (Example.fromdict, Example.fromJSON):


Test patch:
Repository: pytorch/text

Issue:
Multi30k Test Set URL Changed
Looks like the organizers moved the test set to : 

http://www.quest.dcs.shef.ac.uk/wmt17_files_mmt/mmt_task1_test2016.tar.gz


Discussion:

Resolving patch:
diff --git a/torchtext/datasets/translation.py b/torchtext/datasets/translation.py
--- a/torchtext/datasets/translation.py
+++ b/torchtext/datasets/translation.py
@@ -73,7 +73,8 @@ class Multi30k(TranslationDataset):
 
     urls = ['http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz',
             'http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz',
-            'https://staff.fnwi.uva.nl/d.elliott/wmt16/mmt16_task1_test.tgz']
+            'http://www.quest.dcs.shef.ac.uk/'
+            'wmt17_files_mmt/mmt_task1_test2016.tar.gz']
     name = 'multi30k'
     dirname = ''
 


Test patch:
Repository: pytorch/text

Issue:
ValueError: Got input vectors of type <type 'str'>, expected str or Vectors object
when I run the code, it has the error,
**Traceback (most recent call last):
  File "test/example.py", line 16, in <module>
    inputs.vocab.load_vectors('glove.840B.300d')
  File "/Users/wangyuan/anaconda2/lib/python2.7/site-packages/torchtext/vocab.py", line 129, in load_vectors
    "Vectors object".format(type(vector)))
ValueError: Got input vectors of type <type 'str'>, expected str or Vectors object**


Discussion:

Resolving patch:
diff --git a/torchtext/vocab.py b/torchtext/vocab.py
--- a/torchtext/vocab.py
+++ b/torchtext/vocab.py
@@ -114,7 +114,9 @@ def load_vectors(self, vectors):
         if not isinstance(vectors, list):
             vectors = [vectors]
         for idx, vector in enumerate(vectors):
-            if isinstance(vector, six.text_type):
+            if six.PY2 and isinstance(vector, str):
+                vector = six.text_type(vector)
+            if isinstance(vector, six.string_types):
                 # Convert the string pretrained vector identifier
                 # to a Vectors object
                 if vector not in pretrained_aliases:


Test patch:
diff --git a/test/test_vocab.py b/test/test_vocab.py
--- a/test/test_vocab.py
+++ b/test/test_vocab.py
@@ -51,7 +51,7 @@ def test_vocab_download_fasttext_vectors(self):
         # to test string aliases.
         for i in range(3):
             if i == 2:
-                vectors = "fasttext.simple.300d"
+                vectors = str("fasttext.simple.300d")  # must handle str on Py2
             else:
                 vectors = FastText(language='simple')