giseldo/neodataset huggingface dataset:
In adswerve/target-bigquery, state objects are merged

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

`target-bigquery`, default variant `adswerve` merges state objects, causing partition state created by the SDK to be transformed into a format it can not parse.


### What is the expected *correct* behavior?
_What should be happening?_

The target should not manipulate state.

### Steps to reproduce
_How one can reproduce the issue?_

Use a tap with partitioned state (e.g. `MeltanoLabs/tap-github` with `adswerve/target-bigquery` and notice the transformed state object.

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

See: https://gitlab.com/meltano/sdk/-/issues/300+

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_

Default the target's `merge_state_messages` setting to false.

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 1.0
Support `meltano add` directly from Hub

(Probably) blocked by #3031. 

We'd like users to be able to add MeltanoHub plugins directly from the `meltano add` command.

Once this is working, the Hub can be updated with the install instructions on each detail page.

For instance, similar to the `choco install ...` CLI guidance on Chocolatey detail pages:

<details><summary>Click to expand</summary>

![image](/uploads/90016121dc45b9a0d726c8124a7e78ed/image.png)

</details>

... we'd have the text:

> Install with Meltano:
>
> ```console
> meltano add extractor tap-gitlab --variant=MeltanoLabs
> ```

... or with an http ref:

> Install with Meltano:
>
> ```console
> meltano add extractor tap-gitlab --from_ref=https://hub.meltano.com/singer/taps/tap-gitlab--meltanolabs.yml
> ```

We'll likely need to first build #3031+, since this is the issue which decouples us from `discovery.yml`. We could go in another order, but then the inference is that the entire plugin definition would need to live in their `meltano.yml` file, which is parity with `--custom` but doesn't give a similar experience versus `discovery.yml`.

Story points: 8.0
--debug-level=True help function didn't help a new Meltano user

Was just debugging meltano with a new Meltano user. They hit an exception with tap-csv, had the standard "failed while in discovery" which pointing to logs (which they found), and then they tried to run with --log-level=DEBUG but it wasn't clear where to put --log-level=DEBUG so they did something like this

`meltano elt tap-csv target-jsonl --job-id=abc --log-level=DEBUG`

Easy fix is to probably just extend the message a bit to give an example of where to put --log-level? Easy MR to make but not sure if you'll agree!

Story points: 1.0
Flake8 --diff not working as expected from forks

Following from the latest on community-contributed MR !2540, it seems the default branch targeting implemented in !2543 is not working as expected from a fork. Rather than compare `upstream/master` with `forkname/branchname`, the pipeline seems to be comparing `forkname/master` with `forkname/branchname`.

```
$ git fetch origin $CI_DEFAULT_BRANCH; # collapsed multi-line command
From https://gitlab.com/vischous/meltano
 * branch                master     -> FETCH_HEAD
Comparing to master
$ declare FILES=$(git diff --name-only ${CI_COMMIT_SHA} $REF | grep "\.py$")
$ if [ -n "${FILES}" ]; then # collapsed multi-line command
Changed files are\nsrc/meltano/__init__.py
src/meltano/api/app.py
src/meltano/api/controllers/orchestrations.py
src/meltano/api/security/forms.py
src/meltano/cli/__init__.py
src/meltano/cli/add.py
src/meltano/cli/cli.py
...
```

To resolve this, we probably need to somehow explicitly compare with `upstream/master` or a similar reference, and to do so in a way that also works on the main `meltano` repo, which would not have an `upstream` reference.

Cc @edgarrmondragon, @pandemicsyn

Story points: 2.0
Freeze Gitlab docker images as stable, rather than continually updating

Blocks https://gitlab.com/meltano/meltano/-/merge_requests/2542

Before we can direct users to leverage our Gitlab-hosted docker images, we need to resolve a potential issue where `latest*` and `v1.xx.0*` images are apparently being constantly updated on every build to `master`.

Probably the solution is to have a more clear delineation between which images are "dev/test" images needed by later CI steps, and which images are "published/stable" images that can be relied upon by users with production workloads.

Notes:

- We can _optionally_ keep `latest` and `latest*` images linked to master CI builds, with the proper caveat/disclaimer to users that they are non-stable releases for the purposes of previewing unreleased features. (Not for production workloads.)
- Versioned images `*v1.xx.0*` should definitely not be updated continually with new `master` updates. The timestamp on those images reflects them being continually updated as of now.

Story points: 4.0
Full check for broken links in Python text, especially links to docs.meltano.com

As per #3263, it seems we may have a number of undetected 404 errors and/or broken links - especially in Python code which doesn't have any kind of auto-404 detection. A one-time regex or code search audit is probably warranted, as suggested by Derek in the related MR. 

~"urgency::higher"

Story points: 2.0
Update the Getting Started guide in the docs now that environments are created for you

Once !2526 ships, users will no longer need to create their own environments, as documented in the [Getting Started guide](https://docs.meltano.com/getting-started#add-an-environment). This should be updated to just `list` functions, to introduce users to the created environments.

Thanks @pnadolny13 for spotting this 👏

Story points: 2.0
Nested values in array settings don't have environment variables expanded

Related to #2366 

### What is the current *bug* behavior?

I was having trouble using environment variables in the `meltano.yml` file, and it seems they are only supported for top-level configuration options.

If I define `SOME_ENV_VAR=SOME_VALUE` in the `.env` file and I write this configuration yaml:
```yaml
plugins:
  extractors:
    - name: tap-spreadsheets-anywhere
      variant: ets
      pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
      config:
        test: $SOME_ENV_VAR
        tables:
          - path: $SOME_ENV_VAR
            name: my_table
```

Then this is the actual configuration used by meltano:
```
$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "$SOME_ENV_VAR",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}
```

And the `tables.path` variable is not set.



### What is the expected *correct* behavior?

The environment variable `SOME_ENV_VAR` should expend to `SOME_VALUE` in the `tables` array as well, and we should have:
```
$ meltano config tap-spreadsheets-anywhere
{
  "tables": [
    {
      "path": "SOME_VALUE",
      "name": "my_table"
    }
  ],
  "test": "SOME_VALUE"
}
```

### Steps to reproduce

- add the configuration above in `meltano.yml`
- add the line `SOME_ENV_VAR=SOME_VALUE` to your `.env` file
- see the result with `meltano config tap-spreadsheets-anywhere`


### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_



### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Improve plugin command error message to clarify usage syntax

Related to https://gitlab.com/meltano/meltano/-/issues/3237#note_840251651 where @kgpayne said:

>Due to our custom syntax for commands, using the executable name is going to be equally jarring when users get to executing `meltano invoke great_expectations:checkpoint`, which is not functionality of the `great_expectations` executable.

I had a recent experience that I could see tripping people up. I wanted to use `dbt docs generate` and `dbt docs serve` which are not supported as `dbt:x` syntax because its not listed in the [discovery.yml](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/bundle/discovery.yml#L3073). This does not work:

```
meltano invoke dbt:docs generate
```
So I was thinking about adding an extra command to the plugin config like `docs_generate` that runs `docs generate` under the hood but then I realized that this works:

```
meltano invoke dbt docs generate
```

I'm not sure this is a problem but with such a minor difference in syntax I could see it becoming confusing whether your running a meltano command or raw command directly to the package. On one hand its great to allow commands to pass through so all the functionality is preserved but on the other its not clear in what context the command is being run. Maybe just having a warning that a "pass through" command is being used would be helpful, idk but wanted to call it out.

Story points: 2.0
Meltano UI tracking consent "learn more" link not found

The link to "learn more" in our tracking consent popup actually references a link that no longer exists. While it doesn't directly result in a 404 (we redirect to the docs), the link just takes you to the main page. We link to https://docs.meltano.com/reference/settings#send_anonymous_usage_stats

Ref: https://gitlab.com/meltano/meltano/-/blob/master/src/webapp/src/utils/setupToasted.js#L54

Story points: 1.0
Support `--store=keyring` in `meltano config`

As proposed by Derek [in Slack](https://meltano.slack.com/archives/CFG3C3D1Q/p1644431033442989), we could use the [keyring](https://pypi.org/project/keyring/) Python library to store secrets in native OS-level keyrings.

Would add this option alongside others listed in https://docs.meltano.com/reference/command-line-interface#config:

```console
meltano config <plugin> set --store=keyring <name> <value>
```

![image](/uploads/5b404fa0765f20fe78b80c2a5c6daf79/image.png)

Story points: 4.0
Perform API endpoint inventory/discovery

The meltano web API is a bit of an unknown/dark territory for a lot of us. There are quite a few API endpoints that the UI leverages, but they're really not documented at all. There are no API docs, sample requests/responses, expected status codes, etc. We should run a research spike and inventory what API endpoints are available to us so we gain a better understanding of the current state of the API is today.

Objectives for this research spike would be to produce:

- [ ] List of API endpoints/paths roughly documented in this issue for folks to review.
- [ ] Basic summary of what the endpoints do or support (producing a small docstring update MR to add basic method descriptions might be nice).
- [ ] Document any obvious API deficits, or quick wins you may come across.
- [ ] Document any large unknowns in this issue. 
- [ ] Document any other "at first glance" or "next steps" recommendations that might arise.
- [ ] Recommendation for a path forward for auto-generating API docs.

Note from AJ:

Given the short turnaround for our meeting on the 16th (See @tayloramurphy's comment in thread), this probably should to be timeboxed as a `2` (max a few hours investment) for the first iteration.

Story points: 2.0
[Contributed MR]: Allow user to configure Sendgrid asm_group_id

Attached Community-Contributed MR: !2520+

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_



### What is the expected *correct* behavior?
_What should be happening?_



### Steps to reproduce
_How one can reproduce the issue?_
Meltano the org appears to have hardcoded their own Sendgrid account's unsubscribe group, which causes header errors when using Subscriptions with a different Sendgrid account


### Relevant logs and/or screenshots
https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/api/mail.py#L14

### Possible fixes
Allow the use of an environmental variable like `mail_unsubscribe_group`, `asm_group_id` (sendgrid's name for it), `sendgrid_group_id`, etc.



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 2.0
Restore heading anchor links adjacent to headings in the Meltano docs

Looks like the Meltano doc heading anchors are gone/hidden in the new docs site 🤔 They are still available at `<h1>` level by copying links for the table of contents, but it would be great to add anchors next to headings (and subheadings - all of the `<h*>` tags) for linking to specific sections of the docs.

Story points: 4.0
Contribute `GitLabLicenseComplianceReportTable` to `pip-licenses`

This issue is to track contribution of a `GitLabLicenseComplianceReportTable` class to the `pip-licenses` codebase, to allow direct interchangeability between `pip-licenses` and GitLabs `licence_management` tool.

As per [this discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2500#note_820960272), LicenseFinder and the wrapper (called `license_management`) provided by GitLab i) do not support poetry natively and ii) only detect 2/3rds of our total licenses. A brief evaluation of `pip-licenses` detected all but 1 of our dependencies licenses, and supports poetry (by virtue of running within a python virtualenv). However, to adopt `pip-licenses` and still use the License Compliance UI and MR features in GitLab, we need to format the results produced by `pip-licenses` in the same format as the report outputted by the `license_management` wrapper. This can be done by contributing a new report class to `pip-licenses` for optionally outputting the required GitLab artefact.

Story points: 8.0
Meltano Test bug with files bundles

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

This is based on [this slack thread](https://meltano.slack.com/archives/C013EKWA2Q1/p1643388924407829).

### What is the current *bug* behavior?
_What is happening now?_

When I try to use the `test` feature with dbt I get a message saying:
`Command 'test_publish_hub' could not be found. File bundle 'dbt' does not define any commands.`. It looks like its iterating my plugins by name and getting to my file bundle named `dbt` before it gets to my transformer named `dbt`. If I remove the files bundle, my tests work. If I rename my files bundle I get errors.

```
  transformers:
  - name: dbt
    pip_url: git+https://github.com/Tomme/dbt-athena.git@c8e0458f85aaa8f9b693f99903ba95c41b90573b
    config:
      target: athena
    commands:
      test_publish_hub: test --models marts.publish.meltano_hub.*
  files:
  - name: dbt
    pip_url: git+https://gitlab.com/meltano/files-dbt.git@config-version-2
```

### What is the expected *correct* behavior?
_What should be happening?_

The test feature shouldnt consider tests in the files plugin section. Maybe larger than that it might need to accept a plugin.plugin_name combo (i.e. meltano test transformer.dbt:test_123) if a project has multiple plugins with the same name.

### Steps to reproduce
_How one can reproduce the issue?_

Install dbt then add a command starting with the test prefix and run `meltano test dbt:test_name`.


### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_



### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Improve documentation around what a namespace is and does

Reading through the documentation, we don't do a good job of explaining what `namespace` does or means. There's no mental model we're trying to build for the user. When searching for namespace we show the user:

* https://docs.meltano.com/tutorials/postgres-with-postgres#run-meltano-elt
* https://docs.meltano.com/guide/transformation#adding-a-transform-to-your-meltano-project
* https://docs.meltano.com/concepts/project#custom-plugin-definitions
* https://docs.meltano.com/the-project/contributor-guide#plugin-definitions
* https://docs.meltano.com/concepts/plugins#package_name-extra
* https://docs.meltano.com/guide/containerization#gitlab-cicd

But none of those explains what it's doing or why one needs it!

Story points: 2.0
Improve Documentation on Configuring Settings for Custom Plugins

https://docs.meltano.com/guide/configuration#:~:text=If%20we%E2%80%99re%20dealing%20with%20a%20custom%20plugin%20instead

When configuring a custom plugin, the documentation is not clear that the settings configuration needs to be added for Meltano to pick up Environment Variable values for plugin settings. If settings configuration is not added, then plugin config values need to be added directly to meltano.yml to work.

Documentation should clarify that plugin settings need to be configured for Meltano to pickup environmental variables.

Story points: 1.0
Upgrade PIP added to docs

A user of Meltano follows a new path to get started. While I haven't worked with folks who have used the new path from the new website, I have seen folks who are new to Python fail at the `pip install meltano` stage due to pip not being up to date (Every user I've worked with without fail who isn't from Python fails here, and needs my help, had 4 people fail here 1 of which being a dev with>10 year experiance, all of these devs were not python devs) , the error received when this happens is not very clear as it's normally due to some dependency resolution issue that's fixed in newer versions of pip. 

1. https://docs.meltano.com/getting-started
1. https://docs.meltano.com/guide/installation

And anywhere else a new venv is created we should direct people to update their pip version. Thoughts? Easy to update but wanted to be sure you're all onboard

Related to !2480

Story points: 1.0
Support for disabling elt.log creation

The `meltano elt` command will naively write out a log file on every invocation, even in the presence of a logging.yaml config. The elt.log has no built-in support for log rotation/max size/etc and users may not even always be aware of its existence. In situations where disk space is limited, this could cause issues. 

Since we support custom log configs via logging.yaml now, we could give users an option to explicitly disable elt.log generation.

Story points: 4.0
discovery.yml missing from meltano.com

With the update to the new meltano.com marketing we're no longer pushing a discovery.yml and requests return a 404:

```
(melty-3.8) ➜  run-dev2 curl -I https://meltano.com/discovery.yml
HTTP/2 404
server: nginx
date: Wed, 26 Jan 2022 18:03:39 GMT
content-type: text/html; charset=UTF-8
content-length: 71557
vary: Accept-Encoding
vary: Accept-Encoding
expires: Wed, 11 Jan 1984 05:00:00 GMT
link: <https://meltano.com/wp-json/>; rel="https://api.w.org/"
x-powered-by: WP Engine
x-cacheable: non200
cache-control: max-age=600, must-revalidate
x-cache: HIT: 48
x-cache-group: normal
```

Story points: 4.0
Missing output folder

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_
when use the command `meltano init` output folder is not created and when try to use target-jsonl the process fail for that reason



### What is the expected *correct* behavior?

Meltano init command should create the folder



### Steps to reproduce

```bash
$ meltano init example
$ meltano add extract tap-shopify
$ meltano add loader target-jsonl
$ meltano run tap-shopify target-jsonl
```

### Possible fixes

Just create output folder into the init command


### Further regression test

- [ ] Add output folder creation with init command

Story points: 1.0
Problem with `env` key names at the plugin level

Based on a thread from https://gitlab.com/meltano/meltano/-/issues/3173#note_817409586 and from https://gitlab.com/meltano/meltano/-/merge_requests/2489#note_808785679, it would be ideal to be able to set `env` keys at the plugin level vs the current state where it can only be done at the environment level.

The `env` key does seem to work at the plugin level but it prefixes the key in a way that makes it not useful (i.e. `TAP_SLACK__ENV_*`).

Ideally in combination with https://gitlab.com/meltano/meltano/-/issues/3173#note_817409586 we will be able to override environment variables in descending order, where the plugin `env` key is the lowest level and takes precedence. An environment plugin `env` takes precedence over a top level plugin `env`.

@aaronsteers wdyt?

Story points: 4.0