internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: meltano/meltano

Issue:
bug: No way to dismiss image scan alerts
### Meltano Version

NA

### Python Version

NA

### Bug scope

Other

### Operating System

NA

### Description

Currently we use `.github/actions/docker-build-scan-push/check_sarif.py` to analyze the SARIF report created from running `grype` to scan our Docker images. It parses the SARIF JSON file itself to check if there are any issues detected with a severity above some threshold in the range [0.0, 10.0].

Before running this check, we upload the SARIF results to GitHub, which stores them for our repository using the "code scanning" feature. From there, we can review them, dismiss them, and create issues to address them. [An example can be found here](https://github.com/meltano/meltano/security/code-scanning?query=ref%3Arefs%2Fpull%2F6410%2Fmerge+tool%3AGrype).

Our `check_sarif.py` script does not consider whether we've dismissed the issue via GitHub's "code scanning" feature, so we have no way to deem a detected issue acceptable, and have the Docker publish workflow pass. To fix this we should replace `check_sarif.py` with some steps that use [the GitHub code scanning API](https://docs.github.com/en/rest/code-scanning#list-code-scanning-alerts-for-a-repository) to check if there are any issues above some set severity level *that haven't been dismissed*.

### Code

_No response_


Discussion:

Resolving patch:
diff --git a/.github/actions/docker-build-scan-push/check_sarif.py b/.github/actions/docker-build-scan-push/check_sarif.py
deleted file mode 100644
--- a/.github/actions/docker-build-scan-push/check_sarif.py
+++ /dev/null
@@ -1,35 +0,0 @@
-"""Check if the provided SARIF file has any violations at or above some severity level."""
-
-from __future__ import annotations
-
-import argparse
-import json
-
-DEFAULT_SEVERITY_CUTOFF = 4.0
-
-parser = argparse.ArgumentParser()
-parser.add_argument(
-    "sarif_path",
-    help="The path to the SARIF file to be checked.",
-)
-parser.add_argument(
-    "--severity-cutoff",
-    help="Violations with a severity >= this value result in an exit code of 1"
-    + " - must be a number in the range [0.0, 10.0].",
-    type=float,
-    default=DEFAULT_SEVERITY_CUTOFF,
-)
-args = parser.parse_args()
-
-with open(args.sarif_path) as sarif_file:
-    sarif_data = json.load(sarif_file)
-
-first_run = sarif_data["runs"][0]
-triggered_rules = first_run["tool"]["driver"]["rules"]
-
-exit(  # noqa: WPS421
-    any(
-        float(rule["properties"]["security-severity"]) >= args.severity_cutoff
-        for rule in triggered_rules
-    )
-)


Test patch:
Repository: meltano/meltano

Issue:
bug: exit code null on snowplow telemetry
In investigating https://github.com/meltano/internal-data/issues/26 I saw a large increase in bad events. Looks like all of the ones from 2.6.0 and 2.5.0 are from:

`$.exit_code: null found, integer expected`

And I dove in on each one and it's from `add` and `discover` events.

queried using:

```sql
select *
from "RAW"."SNOWPLOW"."EVENTS_BAD"
where date_trunc('week', uploaded_at) > '2022-08-22'
and jsontext ilike '%2.5.0%';
```


Discussion:
FYI @WillDaSilva @aaronsteers 

Resolving patch:
diff --git a/src/meltano/core/tracking/schemas.py b/src/meltano/core/tracking/schemas.py
--- a/src/meltano/core/tracking/schemas.py
+++ b/src/meltano/core/tracking/schemas.py
@@ -30,7 +30,7 @@ def url(self) -> str:
 BlockEventSchema = IgluSchema("block_event", "1-0-0")
 EnvironmentContextSchema = IgluSchema("environment_context", "1-0-0")
 ExceptionContextSchema = IgluSchema("exception_context", "1-0-0")
-ExitEventSchema = IgluSchema("exit_event", "1-0-0")
+ExitEventSchema = IgluSchema("exit_event", "1-0-1")
 PluginsContextSchema = IgluSchema("plugins_context", "1-0-0")
 ProjectContextSchema = IgluSchema("project_context", "1-1-0")
 TelemetryStateChangeEventSchema = IgluSchema("telemetry_state_change_event", "1-0-0")


Test patch:
Repository: meltano/meltano

Issue:
bug: Pytest emits 95+ SQLAlchemy deprecation warnings
### Meltano Version

2.7.1

### Python Version

NA

### Bug scope

Other

### Operating System

N/A

### Description

First seen in https://github.com/meltano/meltano/pull/6795 (test summary: https://github.com/meltano/meltano/actions/runs/3131994616).

### Code

```python
meltano/migrations/versions/6828cc5b1a4f_create_dedicated_state_table.py:335:
SADeprecationWarning: The from_engine() method on Inspector is deprecated and will be removed in a future release. Please use the sqlalchemy.inspect() function on an Engine or Connection in order to acquire an Inspector. (deprecated since: 1.4)
    inspector = Inspector.from_engine(conn)
```



Discussion:

Resolving patch:
diff --git a/src/meltano/migrations/versions/6828cc5b1a4f_create_dedicated_state_table.py b/src/meltano/migrations/versions/6828cc5b1a4f_create_dedicated_state_table.py
--- a/src/meltano/migrations/versions/6828cc5b1a4f_create_dedicated_state_table.py
+++ b/src/meltano/migrations/versions/6828cc5b1a4f_create_dedicated_state_table.py
@@ -13,6 +13,7 @@
 
 import sqlalchemy as sa
 from alembic import op
+import sqlalchemy
 from sqlalchemy import Column, MetaData, func, types
 from sqlalchemy.engine.reflection import Inspector
 from sqlalchemy.ext.declarative import declarative_base
@@ -332,7 +333,7 @@ def upgrade():
     dialect_name = get_dialect_name()
     max_string_length = max_string_length_for_dialect(dialect_name)
     conn = op.get_bind()
-    inspector = Inspector.from_engine(conn)
+    inspector = sqlalchemy.inspect(conn)
     if "state" in inspector.get_table_names():
         op.drop_table("state")
     op.create_table(


Test patch:
Repository: meltano/meltano

Issue:
bug: Can't configure freshdesk-tap
### Meltano Version

2.7.2

### Python Version

3.9

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

Windows 10

### Description

Trying to set up freshdesk tap but getting error.
![image (1)](https://user-images.githubusercontent.com/109072862/196209705-e04c1bc5-78f3-43fb-aa8f-7dc8189fa78a.png)


### Code

_No response_


Discussion:

Resolving patch:
diff --git a/src/meltano/cli/interactive/config.py b/src/meltano/cli/interactive/config.py
--- a/src/meltano/cli/interactive/config.py
+++ b/src/meltano/cli/interactive/config.py
@@ -299,17 +299,28 @@ def configure(self, name, index=None, last_index=None, show_set_prompt=True):
 
     def configure_all(self):
         """Configure all settings."""
+        numeric_choices = [idx for idx, _, _ in self.setting_choices]
+        if not numeric_choices:
+            click.secho(
+                "There are no settings to configure. "
+                "For help, please see https://melta.no#no-plugin-settings-defined",
+                fg="yellow",
+            )
+            self.tracker.track_command_event(CliEvent.completed)
+            return
+
         while True:
             click.clear()
             self._print_home_screen()
-            numeric_choices = [idx for idx, _, _ in self.setting_choices]
-            choices = ["all"] + numeric_choices + ["e"]
+            choices = ["all", *numeric_choices, "e"]
 
             branch = "all"
             try:
                 click.echo()
                 branch = click.prompt(
-                    f"Loop through all settings (all), select a setting by number ({min(int(chs) for chs in numeric_choices)} - {max(int(chs) for chs in numeric_choices)}), or exit (e)?",
+                    "Loop through all settings (all), select a setting by "
+                    f"number ({min(int(chs) for chs in numeric_choices)} - "
+                    f"{max(int(chs) for chs in numeric_choices)}), or exit (e)?",
                     type=click.Choice(choices, case_sensitive=False),
                     default="all",
                     show_choices=False,


Test patch:
Repository: meltano/meltano

Issue:
ci: PyPi publish job fails in "Build distribution" step with error `module 'sqlalchemy' has no attribute 'orm'`
https://github.com/meltano/meltano/actions/runs/3267990463/jobs/5373871668


Discussion:

Resolving patch:
diff --git a/src/meltano/migrations/versions/23ea52e6d784_add_resource_type_to_embed_token.py b/src/meltano/migrations/versions/23ea52e6d784_add_resource_type_to_embed_token.py
--- a/src/meltano/migrations/versions/23ea52e6d784_add_resource_type_to_embed_token.py
+++ b/src/meltano/migrations/versions/23ea52e6d784_add_resource_type_to_embed_token.py
@@ -6,6 +6,7 @@
 
 """
 import sqlalchemy as sa
+import sqlalchemy.orm
 from alembic import op
 
 from meltano.migrations.utils.dialect_typing import (


Test patch:
Repository: meltano/meltano

Issue:
Add `app_id` to meltano telemetry
Currently we only send events to Snowplow from Meltano but soon we'll have several different sources. The app_id so far has been empty when sending from meltano. I think this is usually the way to differentiate the source of the events. Can we add it to the meltano events?

cc @WillDaSilva as discussed in data office hours



Discussion:

Resolving patch:
diff --git a/src/meltano/core/tracking/tracker.py b/src/meltano/core/tracking/tracker.py
--- a/src/meltano/core/tracking/tracker.py
+++ b/src/meltano/core/tracking/tracker.py
@@ -79,10 +79,10 @@ class TelemetrySettings(NamedTuple):
     send_anonymous_usage_stats: bool | None
 
 
-class Tracker:  # noqa: WPS214 - too many methods 16 > 15
+class Tracker:  # noqa: WPS214 - too many methods
     """Meltano tracker backed by Snowplow."""
 
-    def __init__(
+    def __init__(  # noqa: WPS210 - too many local variables
         self,
         project: Project,
         request_timeout: float | tuple[float, float] | None = 3.5,
@@ -121,7 +121,7 @@ def __init__(
             )
 
         if emitters:
-            self.snowplow_tracker = SnowplowTracker(emitters=emitters)
+            self.snowplow_tracker = SnowplowTracker(app_id="meltano", emitters=emitters)
             self.snowplow_tracker.subject.set_lang(locale.getdefaultlocale()[0])
             self.snowplow_tracker.subject.set_timezone(self.timezone_name)
             self.setup_exit_event()


Test patch:
Repository: meltano/meltano

Issue:
bug: Integration tests failing on main
### Meltano Version

N/A

### Python Version

NA

### Bug scope

Other

### Operating System

N/A

### Description

Example failures on `main`:
- https://github.com/meltano/meltano/actions/runs/3534445738
- https://github.com/meltano/meltano/actions/runs/3534480620

Example success on `release/2.10.0`:
- https://github.com/meltano/meltano/actions/runs/3534468951

Affects all integration tests, and may be responsible for failures in the Pytest workflow:
- https://github.com/meltano/meltano/actions/runs/3534001638/jobs/5930358463
- https://github.com/meltano/meltano/actions/runs/3534001638/jobs/5930359021
- https://github.com/meltano/meltano/actions/runs/3534001638/jobs/5930359587 

### Code

_No response_


Discussion:
I have not been able to reproduce this locally yet. Undoing the recent update to the `virtualenv` package had no effect.
Deleting the relevant caches from https://github.com/meltano/meltano/actions/caches also resulted in the tests failing.
I have reproduced the issue inside of a `ubuntu:20.04` Docker container.
The issue stems from running `pip install simplejson==3.11.1` within the plugins' venv. Using `simplejson==3.18.0` (the latest version) works. Version `3.14.0` is the first version that works, which was released back in April of 2018.
Found the source of the problematic dependency: https://github.com/singer-io/singer-python/blob/2c053f4c5f468235bba85a203a252438b1b3c704/setup.py#L15

Related:
- https://github.com/singer-io/singer-python/pull/148
> Found the source of the problematic dependency: [singer-io/singer-python@`2c053f4`/setup.py#L15](https://github.com/singer-io/singer-python/blob/2c053f4c5f468235bba85a203a252438b1b3c704/setup.py#L15)

Part of the reason we removed https://github.com/transferwise/pipelinewise-singer-python as a dependency in the sdk. It had the same issue of pinned dependencies in a library.

@WillDaSilva We used to pin `setuptools` because upgrades used to break installation of many packages, but we changed that recently: https://github.com/meltano/meltano/commit/91ec59ed0a45c09fc75b155c97aed3be49ad7103.
> We used to pin setuptools because upgrades used to break installation of many packages, but we changed that recently: https://github.com/meltano/meltano/commit/91ec59ed0a45c09fc75b155c97aed3be49ad7103.

That'll do it. I'll open a PR to re-add the pin as a short-term fix. Hopefully we can find a better approach for the long-term. Thanks @edgarrmondragon!

Resolving patch:
diff --git a/src/meltano/core/venv_service.py b/src/meltano/core/venv_service.py
--- a/src/meltano/core/venv_service.py
+++ b/src/meltano/core/venv_service.py
@@ -57,7 +57,7 @@ def venv_platform_specs():
         raise Exception(f"Platform {system!r} not supported.") from ex
 
 
-PIP_PACKAGES = ("pip", "setuptools", "wheel")
+PIP_PACKAGES = ("pip", "setuptools==57.5.0", "wheel")
 
 
 class VirtualEnv:


Test patch:
Repository: meltano/meltano

Issue:
Add deprecation warning to `meltano ui` command
Meltano UI is deprecated, and is scheduled for removal in Meltano v3. We should make that known to all who currently use the UI by printing a clear warning message when they run `meltano ui` that states that:
- it is *currently* deprecated
- it will be removed in Meltano v3

Relates to https://github.com/meltano/internal-general/discussions/460

Migration strategies likely fall outside the scope of this issue. From a chat with @aaronsteers on 2022-12-12 the priority will be providing feature-parity with equal-or-better UX via the CLI.

A UI as part of Meltano Cloud (possibly accessible after `meltano login` even if not running workloads on Meltano Cloud) may be available in the future for users who absolutely love UIs and have no interest in moving away from the Meltano UI to the CLI, but that's more or less entirely unplanned, so no promises at this point.

Because we cannot make promises about what we'll be doing to replace the UI, at this point I recommend we keep the deprecation warning minimal and fact-based. Doing so may lead to users asking (many) questions about the impending removal on Slack. This will be a good opportunity for us to discuss with them to figure out what the best path forward will be, i.e. we can ask them why they like the UI, if they'd be happy with the CLI, etc.

Once we've got a more concrete idea for what comes next (and likely after it has been implemented and released), we can update the deprecation warning to advertise it.

@sbalnojan @afolson @tayloramurphy 


Discussion:
Love this - thanks for opening and taking @WillDaSilva 

Resolving patch:
diff --git a/src/meltano/cli/ui.py b/src/meltano/cli/ui.py
--- a/src/meltano/cli/ui.py
+++ b/src/meltano/cli/ui.py
@@ -90,6 +90,13 @@ def ui(ctx, project: Project):
     \b\nRead more at https://docs.meltano.com/reference/command-line-interface#ui
     """
     ctx.obj["project"] = project
+    click.secho(
+        "\n"
+        "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n"
+        "┃ The Meltano UI is deprecated, and scheduled for removal in Meltano 3.0 ┃\n"
+        "┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛\n",
+        fg="red",
+    )
 
 
 @ui.command(cls=InstrumentedCmd, short_help="Start the Meltano UI webserver.")


Test patch:
Repository: meltano/meltano

Issue:
Allow vendor-specific annotations in `meltano.yml`
Discussed in https://github.com/meltano/meltano/discussions/7053

We should update `meltano.schema.json` to permit an object with arbitrary fields (i.e. `"additionalProperties": true`) within any object in the schema that does not already impose restrictions on the permitted additional properties. That is to say:
- If the object has `"additionalProperties": true`, then it can have `annotations`, and no changes are required to the schema.
- If the object has `"additionalProperties": false`, then it can have `annotations` and we must explicitly add it to the properties.
- If the object has `additionalProperties` set to anything else (e.g. limiting it to strings as `env` does), then annotations are not supported, and no change to the schema is necessary.

So the only places in the schema that need to be updated are where it says `"additionalProperties": false`.

Documentation around what this field is for should be added. It should be made clear that the core Meltano library/CLI does not use this field, but it may be used for Meltano Cloud, or third-party tools.

The documentation should also state that we impose no limitation on how it can be used except for requiring that the top-level of each annotation object may only contain objects as properties, and that we recommend that third-party tools keep their annotations under a sensible key, such as the name of the tool.

Testing should be done by adding annotations liberally to integration tests, and ensuring that they are ignored.


Discussion:

Resolving patch:
diff --git a/src/meltano/core/meltano_file.py b/src/meltano/core/meltano_file.py
--- a/src/meltano/core/meltano_file.py
+++ b/src/meltano/core/meltano_file.py
@@ -1,8 +1,9 @@
 """Module for working with meltano.yml files."""
+
 from __future__ import annotations
 
 import copy
-from typing import Iterable
+from typing import Any, Iterable
 
 from meltano.core.behavior.canonical import Canonical
 from meltano.core.environment import Environment
@@ -25,6 +26,7 @@ def __init__(
         environments: list[dict] = None,
         jobs: list[dict] = None,
         env: dict[str, str] = None,
+        annotations: dict[str, dict[Any, Any]] | None = None,  # noqa: WPS442
         **extras,
     ):
         """Construct a new MeltanoFile object from meltano.yml file.
@@ -36,6 +38,7 @@ def __init__(
             environments: Environment configuration for this project.
             jobs: Job configuration for this project.
             env: Environment variables for this project.
+            annotations: Annotations for external tools/vendors - do not access.
             extras: Additional configuration for this project.
         """
         super().__init__(
@@ -47,6 +50,7 @@ def __init__(
             environments=self.load_environments(environments or []),
             jobs=self.load_job_tasks(jobs or []),
             env=env or {},
+            annotations=annotations,
         )
 
     def load_plugins(self, plugins: dict[str, dict]) -> Canonical:


Test patch:
Repository: meltano/meltano

Issue:
bug: environment-level annotations are missing from manifest
### Meltano Version

2.15.3

### Python Version

NA

### Bug scope

Configuration (settings parsing, validation, etc.)

### Operating System

N/A

### Description

Annotations should go from the project files into the manifests without changes, but because the manifests don't contain the environments section the environment annotations are lost.

### Code

_No response_


Discussion:

Resolving patch:
diff --git a/src/meltano/core/manifest/manifest.py b/src/meltano/core/manifest/manifest.py
--- a/src/meltano/core/manifest/manifest.py
+++ b/src/meltano/core/manifest/manifest.py
@@ -368,6 +368,16 @@ def data(self) -> dict[str, Any]:
             for plugin_type, plugins in manifest["plugins"].items()
         }
 
+        # XXX: Meltano shouldn't manipulate annotations, yet here we do
+        # because the alternative is to completely lose the environment-level
+        # annotations.
+        environment = next(iter(manifest["environments"]))
+        if "annotations" in environment:
+            manifest["annotations"] = deep_merge(
+                manifest.get("annotations", {}),
+                environment["annotations"],
+            )
+
         # Everything from the selected environment has been merged into the
         # higher levels, so it can be deleted to avoid any ambiguity about
         # which fields within the manifest should be read.


Test patch:
Repository: meltano/meltano

Issue:
feature: Support arrow key text navigation during interactive config
### Feature scope

CLI (options, error messages, logging, etc.)

### Description

Currently when using interactive config, the arrow keys are interpreted as raw values, rather than as navigation controls:

Examples:

Pressing the up key to try to set the prompt to the last value entered:

```
New value: ^[[A
```

Pressing the left key repeatedly to try to add a missing quote:

```
New value: example"^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D
```

Ideally arrow keys pressed during interactive config would result in typical text navigation behaviour.


Discussion:

Resolving patch:
diff --git a/src/meltano/cli/interactive/config.py b/src/meltano/cli/interactive/config.py
--- a/src/meltano/cli/interactive/config.py
+++ b/src/meltano/cli/interactive/config.py
@@ -2,6 +2,14 @@
 
 from __future__ import annotations
 
+from contextlib import suppress
+
+# NOTE: Importing the readline module enables the use of arrow
+#       keys for text navigation during interactive config.
+#       Refer to https://docs.python.org/3/library/readline.html
+with suppress(ImportError):
+    import readline  # noqa: F401
+
 import click
 from jinja2 import BaseLoader, Environment
 from rich.console import Console, Group


Test patch:
Repository: meltano/meltano

Issue:
bug: `flask` not found when running `meltano-cloud` via `pipx`
### Meltano Version

N/A

### Python Version

3.11

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

Python 3.11 Docker container

### Description

```sh
pip install pipx
pipx install 'git+https://github.com/meltano/meltano.git@cloud#subdirectory=src/cloud-cli'`
pipx ensurepath
bash
meltano-cloud login
```

Results in:
```
FileNotFoundError: [Errno 2] No such file or directory: 'flask'
```

This is because `pipx` only exposes the CLI entrypoints of our package, and not those of its dependencies like `flask`. We can resolve this by using the precise path to the `flask` binary in the active environment rather than searching for it on `$PATH`.

### Code

_No response_


Discussion:

Resolving patch:
diff --git a/src/cloud-cli/meltano/cloud/api/auth/auth.py b/src/cloud-cli/meltano/cloud/api/auth/auth.py
--- a/src/cloud-cli/meltano/cloud/api/auth/auth.py
+++ b/src/cloud-cli/meltano/cloud/api/auth/auth.py
@@ -84,7 +84,11 @@ def callback_server(self) -> t.Iterator[None]:
         server = None
         try:
             server = subprocess.Popen(  # noqa: S607
-                ("flask", "run", f"--port={self.config.auth_callback_port}"),
+                (
+                    str(Path(sys.prefix) / "bin" / "flask"),
+                    "run",
+                    f"--port={self.config.auth_callback_port}",
+                ),
                 env={
                     **os.environ,
                     "FLASK_APP": "callback_server.py",


Test patch: