internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mozmeao/basket

Issue:
Update FxA Created Date in SFMC based on timestamp data in verified events
In #77 we added support for a `createDate` from FxA. They no longer send that field with `verified` events. There are a few other fields that do come with the events that could potentially work. For example, the data looks like this:

```python
{
  'country': 'United States', 
  'countryCode': 'US', 
  'email': 'sync0.3486440592016071@restmail.net', 
  'event': 'verified',
  'iss': 'api-accounts.stage.mozaws.net', 
  'lang': 'en', 
  'locale': 'en', 
  'metricsContext': {
    'flow_time': None, 
    'time': '** ** ** **'
  }, 
  'timestamp': '** ** ** **', 
  'ts': 1582252410.066, 
  'uid': 'dac5bf77929745668ca44ea392e35a4c', 
  'userAgent': 'node-XMLHttpRequest'
}
```

So it looks like `metricsContext['time']`, `timestamp`, or `ts` values could work. I've asked in https://github.com/mozilla/fxa/issues/4138 which would be the best and most reliable one to use. Once we find out we should be able to finish #125.

cc @desiree-mofo


Discussion:

Resolving patch:
diff --git a/basket/news/tasks.py b/basket/news/tasks.py
--- a/basket/news/tasks.py
+++ b/basket/news/tasks.py
@@ -276,7 +276,7 @@ def fxa_verified_sfdc(data):
     email = data['email']
     fxa_id = data['uid']
     lang = data['lang']
-    create_date = data.get('createDate')
+    create_date = data.get('createDate', data.get('ts'))
     newsletters = data.get('newsletters')
     metrics = data.get('metricsContext', {})
     new_data = {


Test patch:
Repository: mozmeao/basket

Issue:
TypeError when SFMC error is returned
SFMC response.content is bytes, but the sfmc.py module tries to concat that message with strings to throw NewsletterExceptions. 


Discussion:
See https://sentry.prod.mozaws.net/operations/basket-prod/issues/7566643/?referrer=slack

Resolving patch:
diff --git a/basket/news/backends/sfmc.py b/basket/news/backends/sfmc.py
--- a/basket/news/backends/sfmc.py
+++ b/basket/news/backends/sfmc.py
@@ -7,6 +7,7 @@
 
 from django.conf import settings
 from django.core.cache import cache
+from django.utils.encoding import force_str
 
 import requests
 from django_statsd.clients import statsd
@@ -89,7 +90,7 @@ def request_token(self, payload):
         try:
             token_response = r.json()
         except ValueError:
-            raise NewsletterException('SFMC Error During Auth: ' + r.content,
+            raise NewsletterException('SFMC Error During Auth: ' + force_str(r.content),
                                       status_code=r.status_code)
 
         if 'accessToken' in token_response:
@@ -102,7 +103,7 @@ def request_token(self, payload):
             del payload['refreshToken']
             return self.request_token(payload)
 
-        raise NewsletterException('SFMC Error During Auth: ' + r.content,
+        raise NewsletterException('SFMC Error During Auth: ' + force_str(r.content),
                                   status_code=r.status_code)
 
     def refresh_token(self, force_refresh=False):
@@ -302,11 +303,11 @@ def send_sms(self, phone_numbers, message_id):
         url = self.sms_api_url.format(message_id)
         response = requests.post(url, json=data, headers=self.auth_header, timeout=10)
         if response.status_code >= 500:
-            raise NewsletterException('SFMC Server Error: {}'.format(response.content),
+            raise NewsletterException('SFMC Server Error: {}'.format(force_str(response.content)),
                                       status_code=response.status_code)
 
         if response.status_code >= 400:
-            raise NewsletterException('SFMC Request Error: {}'.format(response.content),
+            raise NewsletterException('SFMC Request Error: {}'.format(force_str(response.content)),
                                       status_code=response.status_code)
 
     @time_request
@@ -314,11 +315,11 @@ def bulk_upsert_rows(self, de_name, values):
         url = self.rowset_api_url.format(de_name)
         response = requests.post(url, json=values, headers=self.auth_header, timeout=30)
         if response.status_code >= 500:
-            raise NewsletterException('SFMC Server Error: {}'.format(response.content),
+            raise NewsletterException('SFMC Server Error: {}'.format(force_str(response.content)),
                                       status_code=response.status_code)
 
         if response.status_code >= 400:
-            raise NewsletterException(response.content, status_code=response.status_code)
+            raise NewsletterException(force_str(response.content), status_code=response.status_code)
 
 
 sfmc = SFMC()


Test patch:
Repository: mozmeao/basket

Issue:
Connection Reset errors from requests due to long-lived sessions for SFDC
We see a good number of `OSError("(104, 'ECONNRESET')")` errors ([see sentry](https://sentry.prod.mozaws.net/operations/basket-prod/issues/9531465/?query=is%3Aunresolved)) seemingly due to AWS closing our connection to SFDC.

Alan found a workaround in an [issue on the requests lib](https://github.com/psf/requests/issues/4506) that seems like a possible fix. We should try that.


Discussion:

Resolving patch:
diff --git a/basket/news/backends/sfdc.py b/basket/news/backends/sfdc.py
--- a/basket/news/backends/sfdc.py
+++ b/basket/news/backends/sfdc.py
@@ -6,7 +6,10 @@
 
 from django.conf import settings
 from django.core.cache import caches
+from django.utils.functional import cached_property
 
+import requests
+import requests.adapters
 import simple_salesforce as sfapi
 from django_statsd.clients import statsd
 from simple_salesforce.api import DEFAULT_API_VERSION
@@ -315,13 +318,14 @@ def _call_salesforce(self, method, url, **kwargs):
 
 
 class RefreshingSalesforce(RefreshingSFMixin, sfapi.Salesforce):
-    def __init__(self):
+    def __init__(self, session=None):
         self.refresh_session()
         super().__init__(
             session_id=self.session_id,
             instance=self.sf_instance,
             version=DEFAULT_API_VERSION,
             domain=settings.SFDC_SETTINGS["domain"],
+            session=session,
         )
 
 
@@ -347,7 +351,7 @@ def _get_type(self, name):
         if not settings.SFDC_SETTINGS.get("username"):
             return None
 
-        sf = RefreshingSalesforce()
+        sf = RefreshingSalesforce(self._session)
         if name == "sf":
             return sf
         else:
@@ -356,10 +360,17 @@ def _get_type(self, name):
                 session_id=sf.session_id,
                 sf_instance=sf.sf_instance,
                 sf_version=sf.sf_version,
-                session=sf.session,
+                session=self._session,
                 session_expires=sf.session_expires,
             )
 
+    @cached_property
+    def _session(self):
+        session = requests.Session()
+        adapter = requests.adapters.HTTPAdapter(max_retries=4)
+        session.mount("https://", adapter)
+        return session
+
     @property
     def sf(self):
         return self._get_type("sf")


Test patch:
Repository: mozmeao/basket

Issue:
Expand set of fields for PII scrubbing
Based on what I'm seeing, we should add `primary_email` to the list of fields we scrub before sending to Sentry.


Discussion:
I've added this to the Sentry-side list just now

Resolving patch:
diff --git a/basket/settings.py b/basket/settings.py
--- a/basket/settings.py
+++ b/basket/settings.py
@@ -356,6 +356,7 @@ def get_default_gateway_linux():
     "last_name",
     "mobile_number",
     "payee_id",
+    "primary_email",
     "remote_addr",
     "remoteaddresschain",
     "token",


Test patch:
diff --git a/basket/base/tests/test__utils.py b/basket/base/tests/test__utils.py
--- a/basket/base/tests/test__utils.py
+++ b/basket/base/tests/test__utils.py
@@ -43,6 +43,7 @@ def test_pre_sentry_sanitisation__before_send_setup():
         "last_name",
         "mobile_number",
         "payee_id",
+        "primary_email",
         "remote_addr",
         "remoteaddresschain",
         "token",
Repository: mozmeao/basket

Issue:
Add content-type header for Acoustic Transact requests
Transact customers using Oauth for submissions should add header: 

`Content-Type : text/plain` or 
`Content-Type : text/xml`



Discussion:

Resolving patch:
diff --git a/basket/news/backends/acoustic.py b/basket/news/backends/acoustic.py
--- a/basket/news/backends/acoustic.py
+++ b/basket/news/backends/acoustic.py
@@ -90,6 +90,7 @@ def _call(self, xml):
                 self.api_endpoint,
                 data=force_bytes(xml),
                 timeout=10,
+                headers={"Content-Type": "text/xml"},
             )
         except ConnectionError:
             # try one more time
@@ -97,6 +98,7 @@ def _call(self, xml):
                 self.api_endpoint,
                 data=force_bytes(xml),
                 timeout=10,
+                headers={"Content-Type": "text/xml"},
             )
 
         return process_response(response)
@@ -115,6 +117,7 @@ def _call_xt(self, xml):
             self.api_xt_endpoint,
             data=force_bytes(xml),
             timeout=10,
+            headers={"Content-Type": "text/xml"},
         )
         return process_tx_response(response)
 


Test patch:
diff --git a/basket/news/tests/test_acoustic.py b/basket/news/tests/test_acoustic.py
new file mode 100644
--- /dev/null
+++ b/basket/news/tests/test_acoustic.py
@@ -0,0 +1,37 @@
+from unittest.mock import Mock, patch
+
+from django.test import TestCase
+
+from basket.news.backends.acoustic import acoustic, acoustic_tx
+
+
+class TestAcousticBackend(TestCase):
+    @patch("requests_oauthlib.OAuth2Session.post")
+    def test_content_type_xml(self, mock_session_post):
+        """
+        Test that we add the correct content-type headers.
+        """
+        # Need a valid XML response to avoid a parse error.
+        mock_session_post.return_value = Mock(text='<?xml version="1.0"?><test></test>')
+
+        acoustic.send_mailing("mailing-ID", "contact@example.com")
+
+        assert mock_session_post.call_args.kwargs["headers"] == {
+            "Content-Type": "text/xml"
+        }
+
+
+class TestAcousticTxBackend(TestCase):
+    @patch("requests_oauthlib.OAuth2Session.post")
+    def test_content_type_xml(self, mock_session_post):
+        """
+        Test that we add the correct content-type headers.
+        """
+        # Need a valid XML response to avoid a parse error.
+        mock_session_post.return_value = Mock(text='<?xml version="1.0"?><test></test>')
+
+        acoustic_tx.send_mail("contact@example.com", "AAAA")
+
+        assert mock_session_post.call_args.kwargs["headers"] == {
+            "Content-Type": "text/xml"
+        }
Repository: wearepal/EthicML

Issue:
Generate documentation with Sphinx
Sphinx is the standard documentation generator for Python. Our code is using Google-style documentation which [is supported by Sphinx](https://www.sphinx-doc.org/en/1.8/usage/extensions/napoleon.html)!

The best tutorial I've found is this: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/#getting-started


Discussion:

Resolving patch:
diff --git a/docsource/source/conf.py b/docsource/source/conf.py
new file mode 100644
--- /dev/null
+++ b/docsource/source/conf.py
@@ -0,0 +1,71 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath('..'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'EthicML'
+copyright = '2019, O. Thomas, T. Kehrenberg'
+author = 'O. Thomas, T. Kehrenberg'
+
+# The full version, including alpha/beta/rc tags
+release = '0.1.0a3'
+
+
+# -- General configuration ---------------------------------------------------
+autoclass_content = "both"  # include both class docstring and __init__
+autodoc_default_flags = [
+    # Make sure that any autodoc declarations show the right members
+    "members",
+    "inherited-members",
+    "show-inheritance",
+]
+napoleon_numpy_docstring = False  # Force consistency, leave only Google
+napoleon_use_rtype = False  # More legible
+add_module_names = False
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    # Need the autodoc and autosummary packages to generate our docs.
+    'sphinx.ext.autodoc',
+    # 'sphinx.ext.autosummary',
+    # The Napoleon extension allows for nicer argument formatting.
+    'sphinx.ext.napoleon',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'nature'  # 'alabaster'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']


Test patch:
Repository: wearepal/EthicML

Issue:
run_metrics() should not append predictions.info to the results
This should be done somewhere else because not everyone using run_metric wants that behavior.


Discussion:
cc @MylesBartlett 

Resolving patch:
diff --git a/ethicml/evaluators/evaluate_models.py b/ethicml/evaluators/evaluate_models.py
--- a/ethicml/evaluators/evaluate_models.py
+++ b/ethicml/evaluators/evaluate_models.py
@@ -63,7 +63,7 @@ def run_metrics(
     per_sens_metrics: Sequence[Metric] = (),
     diffs_and_ratios: bool = True,
     use_sens_name: bool = True,
-) -> Dict[str, HyperParamValue]:
+) -> Dict[str, float]:
     """Run all the given metrics on the given predictions and return the results.
 
     :param predictions: DataFrame with predictions
@@ -73,8 +73,9 @@ def run_metrics(
     :param diffs_and_ratios: if True, compute diffs and ratios per sensitive attribute (Default: True)
     :param use_sens_name: if True, use the name of the senisitive variable in the returned results.
                         If False, refer to the sensitive varibale as `S`. (Default: True)
+    :returns: A dictionary of all the metric results.
     """
-    result: Dict[str, HyperParamValue] = {}
+    result: Dict[str, float] = {}
     if predictions.hard.isna().any(axis=None):  # type: ignore[arg-type]
         return {"algorithm_failed": 1.0}
     for metric in metrics:
@@ -88,7 +89,6 @@ def run_metrics(
             per_sens.update(diffs_ratios)
         for key, value in per_sens.items():
             result[f"{metric.name}_{key}"] = value
-    result.update(predictions.info)
     return result  # SUGGESTION: we could return a DataFrame here instead of a dictionary
 
 
@@ -333,6 +333,7 @@ def _gather_metrics(
                 **hyperparameters,
             }
             df_row.update(run_metrics(predictions, data_info.test, metrics, per_sens_metrics))
+            df_row.update(predictions.info)
 
             results_df = results_df.append(df_row, ignore_index=True, sort=False)
 


Test patch:
diff --git a/tests/metrics/nonparamaterized_metric_test.py b/tests/metrics/nonparamaterized_metric_test.py
--- a/tests/metrics/nonparamaterized_metric_test.py
+++ b/tests/metrics/nonparamaterized_metric_test.py
@@ -125,8 +125,7 @@ def test_get_info(toy_train_val: TrainValPair):
     train, test = toy_train_val
     model: LRCV = LRCV()
     predictions: Prediction = model.run(train, test)
-    results = em.run_metrics(predictions, test, [], [])
-    assert results["C"] == approx(166.810, abs=0.001)
+    assert predictions.info["C"] == approx(166.810, abs=0.001)
 
 
 def test_tpr_diff_non_binary_race():
Repository: pyscript/pyscript

Issue:
Replace Tailwind with Pure CSS
Tailwind serves the goal of moving fast towards a first goal of having a minimum usable framework in place but it clearly shows its limitations in serving a framework like PyScript. This is mostly due to PyScript being a Framework and not an application. Here are a few of the problems:

* Tailwind is too big for a framework resulting either in a huge chunk of CSS to load 
* Not all users are interested in the visual components and they are currently paying a (high) cost for something they are not using
* PyScript should focus more on the flexibility of integrating well with applications and other frameworks. Tailwind is too invasive in that sense.

The goal is to replace Tailwind with pure CSS and maintain the overall look and feel of the current PyScript components.


Discussion:

Resolving patch:
diff --git a/pyscriptjs/src/pyscript.py b/pyscriptjs/src/pyscript.py
--- a/pyscriptjs/src/pyscript.py
+++ b/pyscriptjs/src/pyscript.py
@@ -262,15 +262,15 @@ def register_parent(self, parent):
 
     def create(self):
         console.log("creating section")
-        new_child = create("section", self._id, "task bg-white my-1")
+        new_child = create("div", self._id, "py-li-element")
         console.log("creating values")
 
         console.log("creating innerHtml")
         new_child._element.innerHTML = dedent(
             f"""
-            <label for="flex items-center p-2 ">
+            <label id="{self._id}" for="flex items-center p-2 ">
               <input class="mr-2" type="checkbox" class="task-check">
-              <p class="m-0 inline">{self.render_content()}</p>
+              <p>{self.render_content()}</p>
             </label>
             """
         )
@@ -304,7 +304,7 @@ def render_content(self):
 
 
 class PyListTemplate:
-    theme = PyWidgetTheme("flex flex-col-reverse mt-8 mx-8")
+    theme = PyWidgetTheme("py-li-element")
     item_class = PyItemTemplate
 
     def __init__(self, parent):


Test patch:
Repository: pyscript/pyscript

Issue:
Python Plugin Methods are Executed Twice
The Python plugin methods appear to be getting called twice each. To recreate, run `make test-integration ARGS='-k test_execution_hooks --headed` and look at the console log. You'll see both `afterSetup` and `afterStartup` are logged twice. (The test passes because it checks that each is present at least once).

This is not just in testing - if you load the PyMarkdown plugin and look at the dev console, you'll see `runtime Received: [object Object]` present twice.

Tested on unstable, on Chromium.


Discussion:

Resolving patch:
diff --git a/pyscriptjs/src/python/pyscript.py b/pyscriptjs/src/python/pyscript.py
--- a/pyscriptjs/src/python/pyscript.py
+++ b/pyscriptjs/src/python/pyscript.py
@@ -497,7 +497,6 @@ def __init__(self, name=None):
 
     def init(self, app):
         self.app = app
-        self.app.plugins.addPythonPlugin(create_proxy(self))
 
     def register_custom_element(self, tag):
         # TODO: Ideally would be better to use the logger.


Test patch:
diff --git a/pyscriptjs/tests/integration/test_plugins.py b/pyscriptjs/tests/integration/test_plugins.py
--- a/pyscriptjs/tests/integration/test_plugins.py
+++ b/pyscriptjs/tests/integration/test_plugins.py
@@ -164,7 +164,7 @@ def test_execution_hooks(self):
         # EXPECT it to log the correct logs for the events it intercepts
         log_lines = self.console.log.lines
         for method in hooks_available:
-            assert f"{method} called" in log_lines
+            assert log_lines.count(f"{method} called") == 1
 
         # EXPECT it to NOT be called (hence not log anything) the events that happen
         # before it's ready, hence is not called
Repository: pyscript/pyscript

Issue:
Py Terminal issues aggregation
### Checklist

- [X] I added a descriptive title
- [X] I searched for other issues and couldn't find a solution or duplication
- [X] I already searched in Google and didn't find any good information or help

### What happened?

We have at least 3 issues in the current PyTerminal:

- [x] errors (on main?) are actually not shown
- [x] escape chars, **only on main**, are not recognized
- [x] the config relative URL is resolved differently between main and worker ... this is probably not strictly related to PyTerminal but it's surely worth fixing this too because otherwise terminals can't easily switch from main to worker and/or vice-versa (see https://github.com/pyscript/polyscript/issues/77)

### What browsers are you seeing the problem on? (if applicable)

_No response_

### Console info

_No response_

### Additional Context

_No response_


Discussion:

Resolving patch:
diff --git a/pyscript.core/src/stdlib/pyscript/magic_js.py b/pyscript.core/src/stdlib/pyscript/magic_js.py
--- a/pyscript.core/src/stdlib/pyscript/magic_js.py
+++ b/pyscript.core/src/stdlib/pyscript/magic_js.py
@@ -8,7 +8,7 @@
 
 
 # allow `from pyscript.js_modules.xxx import yyy`
-class JSModule(object):
+class JSModule:
     def __init__(self, name):
         self.name = name
 


Test patch:
Repository: microsoft/hi-ml

Issue:
Add all items required for making the repository public
Ensure that all files have copyright notices, and that editors are set up to automatically insert them (PyCharm does it correctly on InnerEye)

You must run the following source code analysis tools:
CredScan
CodeQL (Semmle)
Component Governance Detection
The easiest way to run these tools is to add thems in your build pipeline in a Microsoft-managed Azure DevOps account.

For CodeQL, please ensure the following (detailed instructions for CodeQL can be found here):
Select the source code language in the CodeQL task.
If your application was developed using multiple languages, add multiple CodeQL tasks.
Define the build variable LGTM.UploadSnapshot=true.
Configure the build to allow scripts to access OAuth token.
If the code is hosted in Github, create Azure DevOps PAT token with code read scope for dev.azure.com/Microsoft (or ‘all’) organization and set the local task variable System_AccessToken with it. (Note: This only works for YAML-based pipelines.)
Review security issues by navigating to semmleportal.azurewebsites.net/lookup. It may take up to one day to process results.


Discussion:

Resolving patch:
diff --git a/src/health/azure/datasets.py b/src/health/azure/datasets.py
--- a/src/health/azure/datasets.py
+++ b/src/health/azure/datasets.py
@@ -1,3 +1,7 @@
+#  ------------------------------------------------------------------------------------------
+#  Copyright (c) Microsoft Corporation. All rights reserved.
+#  Licensed under the MIT License (MIT). See LICENSE in the repo root for license information.
+#  ------------------------------------------------------------------------------------------
 import logging
 from pathlib import Path
 from typing import List, Optional, Union


Test patch:
Repository: microsoft/hi-ml

Issue:
Make the package tag-line on PyPi more concrete
Javier pointed out that our tagline, _Microsoft Health Intelligence AzureML helpers_, on https://pypi.org/manage/project/hi-ml/releases/ is too generic.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -59,10 +59,12 @@
 # Remove any whitespace and blank lines
 install_requires = [line.strip() for line in install_requires if line.strip()]
 
+description = 'Microsoft Health Intelligence package to elevate and monitor scripts to an AzureML workspace'
+
 setup(
     name='hi-ml',
     version=version,
-    description='Microsoft Health Intelligence AzureML helpers',
+    description=description,
     long_description=long_description,
     long_description_content_type='text/markdown',
     url='https://github.com/microsoft/hi-ml',


Test patch: