internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dbt-labs/dbt-core

Issue:
[Feature] Update `agate` version
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Agate is currently pinned to `agate>=1.7.0,<1.8"`. If it doesn't raise breaking changes, can the version be increased to the latest version of `agate` (currently 1.9.1).

Installing the latest version of `dbt-core` using `conda-forge` on `osx-arm64` was previously not possible because there was no support for `agate`. The feedstock for `agate` [has now been updated with a `noarch` package](https://github.com/conda-forge/agate-feedstock/issues/21) for the latest version (1.9.1), but none of the available candidates on `conda-forge` match the current pin (`agate>=1.7.0,<1.8"`). As a result, I still can't install `dbt-core` using `conda-forge`.

### Describe alternatives you've considered

N/A

### Who will this benefit?

Anyone trying to install `dbt-core` via `conda-forge`.

### Are you interested in contributing this feature?

If it's a simple change to the pin and running some tests, I can do that.

### Anything else?

_No response_


Discussion:
Thanks for reaching out @mjkanji !

As noted in https://github.com/dbt-labs/dbt-core/pull/7368, we put an upper bound on `agate` out of an abundance of caution because there have been breaking changes in past patch/minor versions.

The easiest solution would be if https://github.com/conda-forge/agate-feedstock/issues/21 is resolved by including the relevant versions of `agate`.

But we'd also be willing to look at raising the upper bound so that it includes these more recent versions of `agate` available on [PyPI](https://pypi.org/project/agate/#history):


- 1.9.1 Dec 21, 2023
- 1.9.0 Oct 17, 2023
- 1.8.0 Oct 10, 2023

> If it's a simple change to the pin and running some tests, I can do that.

Yep, to the best of my knowledge, that's what it would involve so I'm going to label this as a `good first issue`. Maybe just update [here](https://github.com/dbt-labs/dbt-core/blob/7e164e3ab723c357cb638ad6c1e1beef19a7fec6/core/setup.py#L52) to be:
```python
        "agate>=1.7.0,<1.10",
```

If you change the upper bound, then the CI tests will run automatically when you open a PR.

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -49,7 +49,7 @@
         # ----
         # dbt-core uses these packages deeply, throughout the codebase, and there have been breaking changes in past patch releases (even though these are major-version-one).
         # Pin to the patch or minor version, and bump in each new minor version of dbt-core.
-        "agate>=1.7.0,<1.8",
+        "agate>=1.7.0,<1.10",
         "Jinja2>=3.1.3,<4",
         "mashumaro[msgpack]>=3.9,<4.0",
         # ----


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Make Memory Usage Reporting More Reliable
At present, dbt uses python's resource.getrusage() function to record the max RSS upon termination. This is not reliable in all of our use cases. Unfortunately this value persists across fork and exec calls on linux, which means that the process which kicks off dbt may set a memory floor that dbt never reaches, obscuring its actual memory usage.

To improve the situation, we can use the /proc/self/status file which should be present on Linux systems, extracting the value of the VmHWM line from that file, which reprsents the high water mark (HWM) of the resident set size. When this file is missing we should fall back to the current implementation.


Discussion:

Resolving patch:
diff --git a/core/dbt/cli/requires.py b/core/dbt/cli/requires.py
--- a/core/dbt/cli/requires.py
+++ b/core/dbt/cli/requires.py
@@ -32,6 +32,7 @@
 from dbt.plugins import set_up_plugin_manager
 from dbt.profiler import profiler
 from dbt.tracking import active_user, initialize_from_flags, track_run
+from dbt.utils import try_get_max_rss_kb
 from dbt.version import installed as installed_version
 from dbt_common.clients.system import get_env
 from dbt_common.context import get_invocation_context, set_invocation_context
@@ -155,7 +156,7 @@ def wrapper(*args, **kwargs):
                         command_wall_clock_time=time.perf_counter() - start_func,
                         process_user_time=rusage.ru_utime,
                         process_kernel_time=rusage.ru_stime,
-                        process_mem_max_rss=rusage.ru_maxrss,
+                        process_mem_max_rss=try_get_max_rss_kb() or rusage.ru_maxrss,
                         process_in_blocks=rusage.ru_inblock,
                         process_out_blocks=rusage.ru_oublock,
                     ),
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -5,6 +5,7 @@
 import itertools
 import json
 import os
+import sys
 from enum import Enum
 from pathlib import PosixPath, WindowsPath
 from typing import (
@@ -386,3 +387,22 @@ def strtobool(val: str) -> bool:
         return False
     else:
         raise ValueError("invalid truth value %r" % (val,))
+
+
+def try_get_max_rss_kb() -> Optional[int]:
+    """Attempts to get the high water mark for this process's memory use via
+    the most reliable and accurate mechanism available through the host OS.
+    Currently only implemented for Linux."""
+    if sys.platform == "linux" and os.path.isfile("/proc/self/status"):
+        try:
+            # On Linux, the most reliable documented mechanism for getting the RSS
+            # high-water-mark comes from the line confusingly labeled VmHWM in the
+            # /proc/self/status virtual file.
+            with open("/proc/self/status") as f:
+                for line in f:
+                    if line.startswith("VmHWM:"):
+                        return int(str.split(line)[1])
+        except Exception:
+            pass
+
+    return None


Test patch:
Repository: dbt-labs/dbt-core

Issue:
Support specifying types for recording
### Housekeeping

- [X] I am a maintainer of dbt-core

### Short description

Support record filtering by types.  

### Acceptance criteria

types can be recorded

### Suggested Tests

none

### Impact to Other Teams

None

### Will backports be required?

No

### Context

Dependent on https://github.com/dbt-labs/dbt-common/issues/140


Discussion:

Resolving patch:
diff --git a/core/dbt/cli/requires.py b/core/dbt/cli/requires.py
--- a/core/dbt/cli/requires.py
+++ b/core/dbt/cli/requires.py
@@ -40,7 +40,12 @@
 from dbt_common.events.helpers import get_json_string_utcnow
 from dbt_common.exceptions import DbtBaseException as DbtException
 from dbt_common.invocation import reset_invocation_id
-from dbt_common.record import Recorder, RecorderMode, get_record_mode_from_env
+from dbt_common.record import (
+    Recorder,
+    RecorderMode,
+    get_record_mode_from_env,
+    get_record_types_from_env,
+)
 from dbt_common.utils import cast_dict_to_dict_of_strings
 
 
@@ -100,13 +105,14 @@ def wrapper(*args, **kwargs):
 
 def setup_record_replay():
     rec_mode = get_record_mode_from_env()
+    rec_types = get_record_types_from_env()
 
     recorder: Optional[Recorder] = None
     if rec_mode == RecorderMode.REPLAY:
-        recording_path = os.environ["DBT_REPLAY"]
-        recorder = Recorder(RecorderMode.REPLAY, recording_path)
+        recording_path = os.environ.get("DBT_RECORDER_FILE_PATH")
+        recorder = Recorder(RecorderMode.REPLAY, types=rec_types, recording_path=recording_path)
     elif rec_mode == RecorderMode.RECORD:
-        recorder = Recorder(RecorderMode.RECORD)
+        recorder = Recorder(RecorderMode.RECORD, types=rec_types)
 
     get_invocation_context().recorder = recorder
 


Test patch:
Repository: mozmeao/basket

Issue:
Rate limit user update
We added rate limiting of subscriptions in [PR 206](https://github.com/mozilla/basket/pull/206). We've now seen a flurry of `RetryError`s reaching max retries due to seemingly over 1000 of them being for the same token and set of newsletters. My only theories on this so far are that we've either 1) got an issue causing retries due to locking to increase in number exponentially, or 2) someone has submitted the same request 1000 times. Based on number 2 being the least unlikely I propose that we add rate limiting on the `user` view, much the same as was added to `subscribe`.



Discussion:

Resolving patch:
diff --git a/news/views.py b/news/views.py
--- a/news/views.py
+++ b/news/views.py
@@ -707,11 +707,18 @@ def update_user_task(request, api_call_type, data=None, optin=False, sync=False)
 
     if api_call_type == SUBSCRIBE and email and data.get('newsletters'):
         # only rate limit here so we don't rate limit errors.
-        if is_ratelimited(request, group='news.views.subscribe',
+        if is_ratelimited(request, group='news.views.update_user_task.subscribe',
                           key=lambda x, y: '%s-%s' % (data['newsletters'], email),
                           rate=EMAIL_SUBSCRIBE_RATE_LIMIT, increment=True):
             raise Ratelimited()
 
+    if api_call_type == SET and token and data.get('newsletters'):
+        # only rate limit here so we don't rate limit errors.
+        if is_ratelimited(request, group='news.views.update_user_task.set',
+                          key=lambda x, y: '%s-%s' % (data['newsletters'], token),
+                          rate=EMAIL_SUBSCRIBE_RATE_LIMIT, increment=True):
+            raise Ratelimited()
+
     if sync:
         statsd.incr('news.views.subscribe.sync')
         if settings.MAINTENANCE_MODE and not settings.MAINTENANCE_READ_ONLY:


Test patch:
diff --git a/news/tests/test_update_user_task.py b/news/tests/test_update_user_task.py
--- a/news/tests/test_update_user_task.py
+++ b/news/tests/test_update_user_task.py
@@ -267,3 +267,15 @@ def test_rate_limit(self):
             self.assert_response_ok(response)
             with self.assertRaises(Ratelimited):
                 update_user_task(request, SUBSCRIBE, data, sync=False)
+
+    def test_rate_limit_user_update(self):
+        """Should raise Ratelimited if token attempts to update same newsletters quickly"""
+        request = self.factory.post('/')
+        data = {'token': 'a@example.com', 'newsletters': 'foo,bar'}
+
+        with patch('news.views.newsletter_slugs') as newsletter_slugs:
+            newsletter_slugs.return_value = ['foo', 'bar']
+            response = update_user_task(request, SET, data, sync=False)
+            self.assert_response_ok(response)
+            with self.assertRaises(Ratelimited):
+                update_user_task(request, SET, data, sync=False)
Repository: mozmeao/basket

Issue:
Increase rate limits for subscribing to newsletter
Hi! Is there any chance we could change [these rate limits](https://github.com/mozmar/basket/blob/56fdb5217702634f8b0d2294ff1ab62fe874c78c/news/views.py#L60), perhaps just doubling both the number and the time period? I definitely understand not wanting to overwhelm the servers, but allowing changing one's mind once sounds reasonable :) And we do want them to receive the newsletter after all!

This would make [this issue](https://github.com/mozilla/fxa-bugzilla-mirror/issues/238) less likely to occur (though other handling will still be put in place for it.).

If a PR for this would be looked at I would be happy to do that - LMK. Thanks!


Increase rate limits for subscribing to newsletter
Hi! Is there any chance we could change [these rate limits](https://github.com/mozmar/basket/blob/56fdb5217702634f8b0d2294ff1ab62fe874c78c/news/views.py#L60), perhaps just doubling both the number and the time period? I definitely understand not wanting to overwhelm the servers, but allowing changing one's mind once sounds reasonable :) And we do want them to receive the newsletter after all!

This would make [this issue](https://github.com/mozilla/fxa-bugzilla-mirror/issues/238) less likely to occur (though other handling will still be put in place for it.).

If a PR for this would be looked at I would be happy to do that - LMK. Thanks!




Discussion:
Hmm. Yeah. We don't really account for people doing a quick sub -> unsub -> sub again dance. I'd be very happy to accept a PR with updated rate limits if you're willing to do that. 
Hmm. Yeah. We don't really account for people doing a quick sub -> unsub -> sub again dance. I'd be very happy to accept a PR with updated rate limits if you're willing to do that. 

Resolving patch:
diff --git a/news/views.py b/news/views.py
--- a/news/views.py
+++ b/news/views.py
@@ -54,10 +54,10 @@
 TOKEN_RE = re.compile(r'^[0-9a-f-]{36}$', flags=re.IGNORECASE)
 IP_RATE_LIMIT_EXTERNAL = getattr(settings, 'IP_RATE_LIMIT_EXTERNAL', '40/m')
 IP_RATE_LIMIT_INTERNAL = getattr(settings, 'IP_RATE_LIMIT_INTERNAL', '400/m')
-# one submission for a specific message per phone number per 10 minutes
-PHONE_NUMBER_RATE_LIMIT = getattr(settings, 'PHONE_NUMBER_RATE_LIMIT', '1/10m')
-# one submission for a set of newsletters per email address per 10 minutes
-EMAIL_SUBSCRIBE_RATE_LIMIT = getattr(settings, 'EMAIL_SUBSCRIBE_RATE_LIMIT', '1/10m')
+# two submissions for a specific message per phone number per 20 minutes
+PHONE_NUMBER_RATE_LIMIT = getattr(settings, 'PHONE_NUMBER_RATE_LIMIT', '2/20m')
+# two submissions for a set of newsletters per email address per 20 minutes
+EMAIL_SUBSCRIBE_RATE_LIMIT = getattr(settings, 'EMAIL_SUBSCRIBE_RATE_LIMIT', '2/20m')
 sync_route = Route(api_token=settings.SYNC_KEY)
 
 


Test patch:
Repository: mozmeao/basket

Issue:
Donation fails if last name is spaces
It tries to process the name as if it's real but then there's nothing:

https://sentry.prod.mozaws.net/operations/basket-prod-us-west/issues/673366/

Just need to strip whitespace.


Discussion:

Resolving patch:
diff --git a/basket/news/tasks.py b/basket/news/tasks.py
--- a/basket/news/tasks.py
+++ b/basket/news/tasks.py
@@ -773,12 +773,14 @@ def process_donation(data):
     get_lock(data['email'])
     # tells the backend to leave the "subscriber" flag alone
     contact_data = {'_set_subscriber': False}
-    if 'first_name' in data:
-        if data.get('first_name'):
-            contact_data['first_name'] = data['first_name']
-        if data.get('last_name'):
-            contact_data['last_name'] = data['last_name']
-    elif data.get('last_name'):
+    first_name = data.get('first_name', '').strip()
+    last_name = data.get('last_name', '').strip()
+    if first_name and last_name:
+        contact_data['first_name'] = first_name
+        contact_data['last_name'] = last_name
+    elif first_name:
+        contact_data['first_name'] = first_name
+    elif last_name:
         names = data['last_name'].rsplit(None, 1)
         if len(names) == 2:
             first, last = names


Test patch:
diff --git a/basket/news/tests/test_tasks.py b/basket/news/tests/test_tasks.py
--- a/basket/news/tests/test_tasks.py
+++ b/basket/news/tests/test_tasks.py
@@ -138,6 +138,25 @@ def test_name_splitting(self, sfdc_mock, gud_mock):
             'last_name': 'Kerabatsos',
         })
 
+    def test_name_empty(self, sfdc_mock, gud_mock):
+        """Should be okay if only last_name is provided and is just spaces.
+
+        https://github.com/mozmeao/basket/issues/45
+        """
+        data = self.donate_data.copy()
+        gud_mock.return_value = None
+        del data['first_name']
+        data['last_name'] = '  '
+        with self.assertRaises(RetryTask):
+            # raises retry b/c the 2nd call to get_user_data returns None
+            process_donation(data)
+        sfdc_mock.add.assert_called_with({
+            '_set_subscriber': False,
+            'token': ANY,
+            'record_type': ANY,
+            'email': 'dude@example.com',
+        })
+
     def test_only_update_contact_if_modified(self, sfdc_mock, gud_mock):
         data = self.donate_data.copy()
         gud_mock.return_value = {
Repository: mozmeao/basket

Issue:
Add support for new fields for incoming data from MoFo Donations
https://docs.google.com/document/d/1o7acbXWxQEdyBlz_48ukGf0RNV7q0LhQLSKB3boxN2s/edit?ts=5bbe6bde


Discussion:
Here's the project board tracking work on the donate site: https://github.com/mozilla/donate.mozilla.org/projects/1

Resolving patch:
diff --git a/basket/news/tasks.py b/basket/news/tasks.py
--- a/basket/news/tasks.py
+++ b/basket/news/tasks.py
@@ -827,12 +827,23 @@ def process_donation_event(data):
         raise
 
 
+# all strings and truncated at 2000 chars
 DONATION_OPTIONAL_FIELDS = {
     'SourceURL__c': 'source_url',
     'Donation_Form_URL__c': 'donation_url',
     'Project__c': 'project',
     'PMT_Subscription_ID__c': 'subscription_id',
 }
+# Add these fields as optional for now as some messages
+# could still come through without them. Mix of string
+# and numerical data.
+DONATION_NEW_FIELDS = {
+    'Donation_Locale__c': 'locale',
+    'Processors_Fee__c': 'transaction_fee',
+    'Net_Amount__c': 'net_amount',
+    'Conversion_Amount__c': 'conversion_amount',
+    'Last_4_Digits__c': 'last_4',
+}
 
 
 @et_task
@@ -896,6 +907,10 @@ def process_donation(data):
     if timestamp:
         donation['CloseDate'] = iso_format_unix_timestamp(timestamp)
 
+    for dest_name, source_name in DONATION_NEW_FIELDS.items():
+        if source_name in data:
+            donation[dest_name] = data[source_name]
+
     for dest_name, source_name in DONATION_OPTIONAL_FIELDS.items():
         if source_name in data:
             # truncate at 2000 chars as that's the max for


Test patch:
diff --git a/basket/news/tests/test_tasks.py b/basket/news/tests/test_tasks.py
--- a/basket/news/tests/test_tasks.py
+++ b/basket/news/tests/test_tasks.py
@@ -276,8 +276,13 @@ def test_charge_disputed(self, sfdc_mock):
 class ProcessDonationTests(TestCase):
     donate_data = {
         'created': 1479746809.327,
+        'locale': 'pt-BR',
         'currency': u'USD',
         'donation_amount': u'75.00',
+        'transaction_fee': 0.42,
+        'net_amount': 75.42,
+        'conversion_amount': 42.75,
+        'last_4': u'5309',
         'email': u'dude@example.com',
         'first_name': u'Jeffery',
         'last_name': u'Lebowski',
@@ -404,6 +409,11 @@ def test_donation_data(self, sfdc_mock, gud_mock):
             'Payment_Type__c': 'Recurring',
             'SourceURL__c': data['source_url'],
             'Project__c': data['project'],
+            'Donation_Locale__c': data['locale'],
+            'Processors_Fee__c': data['transaction_fee'],
+            'Net_Amount__c': data['net_amount'],
+            'Conversion_Amount__c': data['conversion_amount'],
+            'Last_4_Digits__c': data['last_4'],
         })
 
     def test_donation_silent_failure_on_dupe(self, sfdc_mock, gud_mock):
Repository: mozmeao/basket

Issue:
Use MySQL Strict Mode
I noticed the following warning during DB migrations in a Jenkins deployment:


> WARNINGS:
> 
> ?: (mysql.W002) MySQL Strict Mode is not set for database connection 'default'
> 
> 	HINT: MySQL's Strict Mode fixes many data integrity problems in MySQL, such as data truncation upon insertion, by escalating warnings into errors. It is strongly recommended you activate it. See: https://docs.djangoproject.com/en/1.11/ref/databases/#mysql-sql-mode


Seems like a good idea to do that.


Discussion:

Resolving patch:
diff --git a/basket/settings.py b/basket/settings.py
--- a/basket/settings.py
+++ b/basket/settings.py
@@ -55,7 +55,10 @@ def path(*args):
                       cast=dj_database_url.parse),
 }
 if DATABASES['default']['ENGINE'] == 'django.db.backends.mysql':
-    DATABASES['default']['OPTIONS'] = {'init_command': 'SET storage_engine=InnoDB'}
+    DATABASES['default']['OPTIONS'] = {
+        'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
+    }
+
 
 CACHES = {
     'default': config('CACHE_URL',


Test patch:
Repository: mozmeao/basket

Issue:
last_name data resolving as NoneType
We recently started getting data from MoFo that's throwing errors when attempting to pull `last_name` from the data.

[This line](https://github.com/mozmeao/basket/blob/prod/basket/news/backends/sfdc.py#L332): `if not data.get('last_name', '').strip():`

Is resulting in this error: `AttributeError: 'NoneType' object has no attribute 'strip'`

Related: MoFo [pushed a code change on 6/6 defaulting names to `None`](https://github.com/mozilla/foundation.mozilla.org/commit/10d2a12881a237cee7baaaf9cbb319715e7b8363#diff-d66144df8fec88b46d1f5e779b75122fR93). This is likely what revealed the bug.
last_name data resolving as NoneType
We recently started getting data from MoFo that's throwing errors when attempting to pull `last_name` from the data.

[This line](https://github.com/mozmeao/basket/blob/prod/basket/news/backends/sfdc.py#L332): `if not data.get('last_name', '').strip():`

Is resulting in this error: `AttributeError: 'NoneType' object has no attribute 'strip'`

Related: MoFo [pushed a code change on 6/6 defaulting names to `None`](https://github.com/mozilla/foundation.mozilla.org/commit/10d2a12881a237cee7baaaf9cbb319715e7b8363#diff-d66144df8fec88b46d1f5e779b75122fR93). This is likely what revealed the bug.


Discussion:

Resolving patch:
diff --git a/basket/news/backends/sfdc.py b/basket/news/backends/sfdc.py
--- a/basket/news/backends/sfdc.py
+++ b/basket/news/backends/sfdc.py
@@ -329,7 +329,7 @@ def add(self, data):
         @param data: user data to add as a new contact.
         @return: None
         """
-        if not data.get('last_name', '').strip():
+        if not (data.get('last_name', '') or '').strip():
             data['last_name'] = LAST_NAME_DEFAULT_VALUE
         self.contact.create(to_vendor(data))
 


Test patch:
Repository: mozmeao/basket

Issue:
update middleware to use k8s values (instead of deis)
related to mozmeao/infra#1104


Discussion:

Resolving patch:
diff --git a/basket/news/middleware.py b/basket/news/middleware.py
--- a/basket/news/middleware.py
+++ b/basket/news/middleware.py
@@ -32,9 +32,9 @@ def process_view(self, request, view_func, view_args, view_kwargs):
 
 class HostnameMiddleware(object):
     def __init__(self, get_response):
-        values = [getattr(settings, x) for x in ['HOSTNAME', 'DEIS_APP',
-                                                 'DEIS_RELEASE', 'DEIS_DOMAIN']]
-        self.backend_server = '.'.join(x for x in values if x)
+        values = [getattr(settings, x) for x in [
+            'CLUSTER_NAME', 'K8S_NAMESPACE', 'K8S_POD_NAME']]
+        self.backend_server = '/'.join(x for x in values if x)
         self.get_response = get_response
 
     def __call__(self, request):
diff --git a/basket/settings.py b/basket/settings.py
--- a/basket/settings.py
+++ b/basket/settings.py
@@ -276,19 +276,19 @@ def get_default_gateway_linux():
 
 
 HOSTNAME = platform.node()
-DEIS_APP = config('DEIS_APP', default=None)
-DEIS_DOMAIN = config('DEIS_DOMAIN', default=None)
-DEIS_RELEASE = config('DEIS_RELEASE', default=None)
+CLUSTER_NAME = config('CLUSTER_NAME', default=None)
+K8S_NAMESPACE = config('K8S_NAMESPACE', default=None)
+K8S_POD_NAME = config('K8S_POD_NAME', default=None)
 
 RAVEN_CONFIG = {
     'dsn': config('SENTRY_DSN', None),
-    'site': '.'.join(x for x in [DEIS_APP, DEIS_DOMAIN] if x),
+    'site': '.'.join(x for x in [K8S_NAMESPACE, CLUSTER_NAME] if x),
     'release': config('GIT_SHA', None),
 }
 
 STATSD_HOST = config('STATSD_HOST', get_default_gateway_linux())
 STATSD_PORT = config('STATSD_PORT', 8125, cast=int)
-STATSD_PREFIX = config('STATSD_PREFIX', DEIS_APP)
+STATSD_PREFIX = config('STATSD_PREFIX', K8S_NAMESPACE)
 STATSD_CLIENT = config('STATSD_CLIENT', 'django_statsd.clients.null')
 
 LOGGING = {


Test patch: