giseldo/neodataset huggingface dataset:
Melano cache error noise after a tap-postgres elt

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_



### What is the expected *correct* behavior?
_What should be happening?_



### Steps to reproduce

Run a postgres to redshift ELT with key-based replication.



### Relevant logs and/or screenshots
![image](/uploads/f3ca4b8681a2bee49e4f0b36dabe3d5e/image.png)


### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Incorrect docs link after `meltano add transformer dbt`

After installing dbt, the docs link provided is `https://meltano.com/docs/transforms.md`. The correct link is: `https://meltano.com/docs/transforms.html`


```
ajsteers@ajs-macbook-pro meltano % meltano add transformer dbt
Transformer 'dbt' already exists in your Meltano project
To add it to your project another time so that each can be configured differently,
add a new plugin inheriting from the existing one with its own unique name:
        meltano add transformer dbt--new --inherit-from dbt

Installing transformer 'dbt'...
Installed transformer 'dbt'


To learn more about transformer 'dbt', visit https://meltano.com/docs/transforms.md
```

Story points: 1.0
Follow-up from "building a custom extractor" tutorial docs

Logging follow-ups for further improvement on the tutorial page:


The following discussions from !2107 should be addressed:

- [ ] @DouweM started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2107#note_558219557):  (+1 comment)

    > Can we refer to the documentation on where this script comes from?

- [ ] @DouweM started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2107#note_558219582): 

    > Like above, I'd rather recommend they use `meltano config <plugin> set password <value>` which will automatically store it in the most appropriate location, which is `.env` in this case.
    > 
    > We don't need to explicitly talk about environment variables then, just like we don't in https://meltano.com/docs/getting-started.html#configure-the-extractor. They can learn more about those if they dive deeper into https://meltano.com/docs/configuration.html and find https://meltano.com/docs/configuration.html#configuring-settings.

- [ ] @DouweM started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2107#note_558219583): 

    > I think we can remove this section entirely, or in favor of a link to https://meltano.com/docs/configuration.html#configuration-layers.
    > 
    > In general, I think this guide should not focus too much on the different ways/places config can be stored; that's something for the Configuration guide.

- [ ] @DouweM started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2107#note_558219587): 

    > Instead of just mentioning a command here in a vacuum, I think we should link to the canonical guide at https://meltano.com/docs/getting-started.html#select-entities-and-attributes-to-extract.
    > 
    > I'm starting to think that this guide covers a lot more than it needs to; basically all of https://meltano.com/docs/getting-started.html still applies, except for some bits under https://meltano.com/docs/getting-started.html#add-an-extractor-to-pull-data-from-a-source. I think it'd be better to only cover on what's custom-extractor-specific, and link to the other guides for everything else.

- [ ] @DouweM started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2107#note_558219590):  (+1 comment)

    > I think this would make more sense as a dedicated doc in the SDK repo that we could link to from here.

Story points: 2.0
Add `--help `docs for meltano CLI

Several of the CLI commands are not documented in `meltano --help`. Adding descriptions here could help streamline the new user experience.

![image](/uploads/d91d5e25b2e7306ba9aa9e9515f3ef9b/image.png)

Also, many of the descriptions are trailing off in the screenshot above '...'.

CLI documentation for reference: https://meltano.com/docs/command-line-interface.html

Reported in slack: https://meltano.slack.com/archives/C01TCRBBJD7/p1618932986365900?thread_ts=1618925301.362900&cid=C01TCRBBJD7

## Update 2021-11-30

This is much better now but still has some missing hints.

```console
$ meltano --help
Usage: meltano [OPTIONS] COMMAND [ARGS]...

  Get help at https://www.meltano.com/docs/command-line-interface.html

Options:
  --log-level [debug|info|warning|error|critical]
  -v, --verbose
  --environment TEXT              Meltano environment name
  --version                       Show the version and exit.
  --help                          Show this message and exit.

Commands:
  add          Add a plugin to your project.
  config       Display Meltano or plugin configuration.
  discover
  elt          meltano elt EXTRACTOR_NAME LOADER_NAME extractor_name: Which...
  environment  Manage Environments.
  init         Creates a new Meltano project
  install      Installs all the dependencies of your project based on the...
  invoke       Invoke the plugin's executable with specified arguments.
  remove       Remove a plugin from your project.
  schedule     Manage Schedules.
  schema
  select       Execute the meltano select command.
  ui
  upgrade
  user
```

Story points: 2.0
Fail when plugin `pip_url` changed and installation is outdated (pip_url drift)

### Problem to solve

I installed Meltano with tap-mysql and target-postgres. After creating some pipelines I realised that for a data set I needed to track deletions and this would require the transferwise variant. I switched the variant in the meltano.yml, but then spent a whilst trying to work out why it was not creating _sdc_deleted_at columns in postgres.

This is partially as I'm new, however the solution was to reinstall target-postgres with the transferwise variant. Everything then just worked, as you'd expect!


### Proposal

It would really help (newer) users if the configuration validated the variant installed when it tried to run them.


### What does success look like, and how can we measure that?

If the installed variant of a tap or a target does not match the variant in the meltano.yml then an error or warning should be shown.

### Proposed spec (added 2022-03-03)

1. During install we drop a marker file into the root of the venv with text of the `pip_url` used during installation. (Or a hash, if we think the pip_url is sensitive.)
   - E.g. `.meltano/.../dbt/meltano_pip_url_marker.txt`
2. During execution we check the marker file against the current `pip_url` value.
3. If the comparison fails, we either warn or throw a hard failure before executing the plugin.

Story points: 4.0
update default dbt version to 0.18.2

This issue is being created to document an MR already in progress.

At this moment, meltano uses `dbt==0.16.1` for the dbt transformer.  There appear to be some installation issues with pinned version of dbt related to `agate` and `PyICU` dependencies.

dbt recently released https://github.com/fishtown-analytics/dbt/releases/tag/v0.18.2, which pins `agate<1.6.2` to avoid installation errors relating to its new dependency `PyICU`.  

Testing was successful installing `dbt==0.18.2` from meltano:

```
 transformers:
  - name: dbt
    # pip_url: dbt==0.16.1
    pip_url: dbt==0.18.2
```


Note there are significant differences in parsing dbt_project files between versions. Migration guide here: https://docs.getdbt.com/docs/guides/migration-guide/upgrading-to-0-17-0

Story points: 12.0
Include discovery output in `meltano elt` output

### What is the current *bug* behavior?

When running `meltano --log-level=debug elt tap-spreadsheets-anywhere target-postgres --job_id=test`, with the tap trying to load an unsupported JSON file, error messages from the tap's discover code are not shown in the output.

After checking with @DouweM, it is apparently related to the fact that meltano elt runs the tap twice, once with --discover to generate the catalog, and then in sync mode along with the target. meltano elt only forwards stdout and stderr from that sync mode run, not the discovery run, meaning that any log messages that only show up during discovery mode are lost, unless discovery mode fails entirely and all output is dumped.

### What is the expected *correct* behavior?

Log messages from the discovery step should be forwarded to the output as well.

### Steps to reproduce

- setup `tap-spreadsheets-anywhere` to load a file incorrectly formatted, so that discovery will fail with an error message. The following config for the tap can be used to trigger such an error:
```yaml
plugins:
  extractors:
  - name: tap-spreadsheets-anywhere
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
        - path: https://salsa.debian.org/iso-codes-team/iso-codes/-/raw/main/data/
        format: json
        name: iso_countries
        pattern: "iso_3166-1.*"
        key_properties:
        - 'alpha_2'
        start_date: '2011-01-01T00:00:00Z'
```
- run `meltano --log-level=debug elt tap-spreadsheets-anywhere target-postgres --job_id=test`
- note that the output does not include the expected error messages (`ERROR Unable to write Catalog entry for 'iso_countries' - it will be skipped due to error 'str' object has no attribute 'items'`) should appear in the logs

### Relevant logs and/or screenshots

### Possible fixes




### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 8.0
Use latest Python 3.9 in CI

[We had to pin Python 3.9 to `3.9.6`](https://gitlab.com/meltano/meltano/-/issues/2913) in our test matrix due to problems with dbt in that version. That makes our `3.9` docker images come with Python `3.9.6`.

This was later fixed upstream in [dbt's `0.20.2` release](https://github.com/dbt-labs/dbt-core/releases/tag/v0.20.2). That means we can go back to using the latest, `3.9.13` at the time of writing.

Story points: 2.0
`tap-quickbooks` should have `properties` capability instead of `catalog`

- Came up recently in Slack: https://meltano.slack.com/archives/CMN8HELB0/p1652851641112949
- The tap uses `--properties`: https://github.com/hotgluexyz/tap-quickbooks/blob/9a9990601f02e91d9ef1357b91b69624eb9dab2b/tap_quickbooks/__init__.py#L272
- `discovery.yml` says it uses `--catalog`: https://gitlab.com/meltano/meltano/-/blob/6d96c9bc9ec5610d87dc01d568bbdb3b162210a4/src/meltano/core/bundle/discovery.yml#L1724

Story points: 1.0
Telemetry project counts inflated if using a custom project_id

@edgarrmondragon helpfully pointed me to this block in our [tap-gitlab conversation](https://github.com/MeltanoLabs/tap-gitlab/pull/77#discussion_r875020636):

https://gitlab.com/meltano/meltano/-/blob/4c8cbc0bdb4db0248a378017c3615d88e8cf1878/src/meltano/core/tracking/ga_tracker.py#L84-95

That block apparently fails initialization of project ID _and creates a new one_ if the project ID is not a valid UUID. In my experience, this was getting written back to my project file and then I would continually revert it and it would reset again, etc. In CI, this probably gets a distinct project ID on each run - meaning each run through CI (and each job) looks like a distinct project.

This is very likely creating a significant inflation of our total domain project IDs and I think we should address as ~"urgency::high". (The code was last touched 12+ months ago, or so, so I imaging this has been an issue for quite a while.)

cc @tayloramurphy, @pnadolny13, @edgarrmondragon, @DouweM

Story points: 2.0
Use `settings_service.feature_flag` convention for lockfiles

The following discussion from !2596 should be addressed:

- [ ] @cjohnhanson started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2596#note_945226196): 

    > Optional follow-up: once !2643 is merged, update this to follow the new `settings_service.feature_flag` convention OR remove this flag parameter and use the feature flag within `ProjectAddService` instead.

Story points: 4.0
Add job support to schedules and to airflow

In https://gitlab.com/meltano/meltano/-/issues/2924 we're adding support for top level jobs concept in meltano.yaml along with adding support for jobs expansion to `meltano run`. We also want to add job support to the `meltano schedule` command, and likely also alter how airflow interacts with schedules ala https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py. 

## Some clarification on things that caused confusion in #2924.

1. We are NOT deprecating support `elt` schedules in favor of `run`, and `meltano schedule` including `meltano schedule run` will support both.
2. Airflow calls `meltano schedule run` [per schedule](https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py#L99), it does NOT use `meltano elt` directly,  `meltano schedule run` uses MeltanoInvoker to spawn a `meltano elt` subprocess.
   - at a minimum `meltano schedule run` will need to support calling both `meltano elt` and `meltano run` based on the scheduled its being asked to run.

## Latest `meltano schedule` UX spec
(ala @tayloramurphy in https://gitlab.com/meltano/meltano/-/issues/2924#note_937990262)

```
meltano schedule <schedule_name> <job_name> <interval> # This is the default
meltano schedule job add <schedule_name> <job_name> <interval> # This is the same behavior as the above but [job] and [add] are default
meltano schedule elt <schedule_name> <extractor> <loader> <interval> [--transform={run,skip,only}] # new syntax for elt workflows Q: Does this break anything in the UI?
meltano schedule elt add <schedule_name> <extractor> <loader> <interval> [--transform={run,skip,only}] # same as above with the default add
meltano schedule <schedule_name> set <job_name> <interval> <environment>
meltano schedule <schedule_name> set name=<job_name> # this is different from the description b/c of the key (same for the next 2) - Q: Is the key required?
meltano schedule <schedule_name> set interval=<interval>
meltano schedule <schedule_name> set environment=<environment>
meltano schedule list [--format=json] Does this work with jobs that are scheduled via the job definition?
meltano schedule run <schedule_name>
```

Story points: 8.0
`meltano config airflow` fails for newly install Airflow plugin

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

When running `meltano config airflow` to print the current Airflow configuration, the command fails with the following error:

```bash
[Errno 2] No such file or directory: '/Users/kp/Projects/squared/data/.meltano/run/airflow/airflow.cfg'
```

Once `meltano invoke airflow webserver` has been run once, the missing file is created and the command succeeds in printing the current config.

### What is the expected *correct* behavior?
_What should be happening?_

`meltano config airflow` should be able to print the current Airflow configuration regardless of the existence of Meltano-managed files.


### Steps to reproduce
_How one can reproduce the issue?_

```bash
meltano init airflow_bug

meltano add orchestrator airflow

meltano install

meltano config airflow
```

I am somewhat surprised that `meltano install` is necessary as `meltano config airflow` is just supposed to print the current config 🤔

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_


### Workaround

As a workaround, a user can run `meltano invoke airflow webserver`, which should fully initialize the airflow resources and prevent the error from triggering.


### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Document and lint against type annotation standards

As we progressively type-annotate our codebase, we should document our type-related standards and expectations in our [Dev Standards](https://handbook.meltano.com/engineering/dev-standards).

If possible, we should also add a flake8 plugin to lint against these standards as part of pre-commit and CI.

This will help new engineers onboard faster and accelerate our progress towards being able to do static type checking as part of our dev and CI workflow.

Related to [this discussion on a recent MR](https://gitlab.com/meltano/meltano/-/merge_requests/2596#note_932443415).

CC @pandemicsyn @edgarrmondragon

Story points: 8.0
"move" and "copy" subcommands for "meltano state"

This is a follow-up from: https://gitlab.com/meltano/meltano/-/issues/2754

We need to add the following subcommands to `meltano state`:
1. `copy`
   * `meltano state copy <JOB-ID-1> <JOB-ID-2>`
1. `move`
   * `meltano state move <JOB-ID-1> <JOB-ID-2>` (or `meltano state rename`)

Story points: 4.0
Make setting-level `env:` declaration a one-way injection into the plugin execution context

Follows from #3431: We want the `env:` usage pattern to be fully unambiguous. The use case this exists for is so that plugins can receive injected environment variable values from Meltano.

1. The user sets the setting value in any supported way.
2. The plugin gets that setting value from the `env:`.
3. If an env var is already set by that name, its value will be replaced by the setting value.


For example: if the OS context has an env var set as `DEBUG_MODE=1` but my plugin has a setting with `env: DEBUG_MODE` and setting value of `0`, then the plugin when invoked will have `DEBUG_MODE=0`, which is the value that the plugin is intended to receive by the Meltano user. ~~Any other env vars from context will be ignored _unless_ they are referenced under `value:`, `env_aliases:`, or the `config:` entry.~~ Any other env vars from context will be ignored _**when initializing the plugins settings**_ _unless_ they are referenced under `value:`, `env_aliases:`, or the `config:` entry.

Story points: 4.0
Fail adding `transform` plugin if no `transformer` is added

Discovered from my dogfooding project (https://gitlab.com/meltano/meta/-/issues/234+), we currently auto-add the `model`, `transform`, and `transformer` plugins when adding extractors.

When adding extractors, we should not auto-add any other plugins:

- `model` plugins are being deprecated.
- `transform` plugins are being deprecated.
- `transformer` plugin can not be inferred to be `dbt`, since we are opting for adapter-specific transformer plugins now.

Story points: 4.0
Allow `--custom` when adding transformers

I discovered in my dogfooding project (https://gitlab.com/meltano/meta/-/issues/234), that adding `--custom` transformers is explicitly disallowed as of now and this is problematic for the path we are going with #3298.

We should allow `transformer` plugins to be defined via `--custom` as we do today for `extractor` plugins and other types.

Story points: 4.0
metadata extra not honored by inherited plugin


### What is the current *bug* behavior?
If the metadata extra is defined in the base configuration of an inherited plugin it is not respected - the metadata from the base configuration of the parent plugin is used instead.



### What is the expected *correct* behavior?
The metadata extra from the inherited plugin definition should be used.



### Steps to reproduce
Using the attached [meltano.yml](/uploads/3ddeb9658b37693e2edfe65519e970c2/meltano.yml) run

`meltano --environment=prod run tap-mysql2 target-jsonl`

SomeTable2 will get replicated but with replication-method=LOG_BASED. If instead you run

`meltano --environment=dev run tap-mysql2 target-jsonl`

SomeTable2 will be replicated using replication-method=FULL_TABLE as expected. The point being that this appears to be a problem specifically with metadata defined on the base inherited plugin only.

Note also that if you change the metadata in the base parent plugin to not use a wildcard, i.e.

```   
    metadata:
      SchemaName-SomeTable:
        replication-method: LOG_BASED
```

The pipeline will fail with an error about the replication method not being defined for SomeTable2. I think that's relevant because it shows that the wildcard isn't overriding the inherited plugin configuration.

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
meltano add transformer dbt - extra lines in meltano.yml

After running `meltano add transformer dbt` the `meltano.yml` related config looks like this

```yml
  transformers:
  - name: dbt
    pip_url: 'dbt-core~=1.0.0 dbt-postgres~=1.0.0 dbt-redshift~=1.0.0 dbt-snowflake~=1.0.0
      dbt-bigquery~=1.0.0

      '
  files:
  - name: dbt
    pip_url: git+https://gitlab.com/meltano/files-dbt.git@config-version-2
```

Note the 2/3 extra new lines

Story points: 1.0
Meltano UI pipelines tab broken since 1.99

Pipelines tab gets stuck in a loading loop erroring out with:

```
TypeError: Cannot read properties of undefined (reading 'includes')
    at app.4c6077ae.js:1:181845
    at Array.map (<anonymous>)
    at i.cloneDeepPipelines (app.4c6077ae.js:1:181810)
    at nr.get (chunk-vendors.43453e12.js:7:30685)
    at nr.evaluate (chunk-vendors.43453e12.js:7:31686)
    at i.cloneDeepPipelines (chunk-vendors.43453e12.js:7:33379)
    at i.ys (app.4c6077ae.js:1:170909)
    at i.e._render (chunk-vendors.43453e12.js:7:23734)
    at i.r (chunk-vendors.43453e12.js:7:27825)
    at nr.get (chunk-vendors.43453e12.js:7:30685)
at @ chunk-vendors.43453e12.js:7
3app.4c6077ae.js:1 
```

Reason is this line here: https://gitlab.com/meltano/meltano/-/blob/master/src/webapp/src/views/Pipelines.vue#L44

We have schedules with unset `interval` keys, as we don't use Meltano as an orchestrator. Because that line tries to perform an `includes` check on an unset key, it causes a TypeError.

Story points: 2.0