giseldo/neodataset huggingface dataset:
meltano run , multiple dbt commands leads to wrong dbt commands running

To reproduce run this
1. `meltano run dbt:compile dbt:run` 

Meltano will run `dbt:run` twice


My actual use case is
1. `meltano run dbt:drop_source_table tap-abc target-postgres dbt:run` 

Noticed run was happening twice, was pretty hard to notice this was happening.

Story points: 2.0
meltano init not creating DB

This happens on windows, I'd assume not other OS's, the DB is not initting. `meltano, version 1.99.0` I'm not certain this is new behavior it could have been around all this time and I just missed putting an issue in (seems unlikely but it's possible) 

1. Run `meltano init test`
1. See output below
```
PS C:\code> meltano --log-level=debug init test-issue
Created test
Creating project files...
  test/
   |-- meltano.yml
   |-- README.md
   |-- requirements.txt
   |-- output\.gitignore
   |-- .gitignore
   |-- model\.gitkeep
   |-- extract\.gitkeep
   |-- load\.gitkeep
   |-- transform\.gitkeep
   |-- analyze\.gitkeep
   |-- notebook\.gitkeep
   |-- orchestrate\.gitkeep
Creating system database...2022-04-08T19:54:04.823778Z [info     ] DB connection failed. Will retry after 5s. Attempt 1/3
2022-04-08T19:54:09.841274Z [info     ] DB connection failed. Will retry after 5s. Attempt 2/3
2022-04-08T19:54:14.857067Z [info     ] DB connection failed. Will retry after 5s. Attempt 3/3
2022-04-08T19:54:19.872743Z [error    ] Could not connect to the Database. Max retries exceeded.
(sqlite3.OperationalError) unable to open database file
(Background on this error at: https://sqlalche.me/e/14/e3q8)

PS C:\code> meltano --version
meltano, version 1.99.0
```

Story points: 2.0
Incorrect `get` behaviour with default settings overriding environment-specific plugin settings

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?

We have incorrect resolution order between default values at the plugin setting level (such as from `discovery.yml`) versus the settings specified in an environment.

If an environment setting is provided, that should take precedent over default values, which is not the case as of now.

This does affect plugin settings at the top-level `plugins` definitions (perhaps only in certain situations though?).

Using `set` to set the value of a plugin extra to a non-default value uses the `default_environment`, however using `set` to set the value _back_ to a default value (explicitly) does not remove the setting override from the default environment.

### What is the expected *correct* behavior?
_What should be happening?_

`set` should work exactly the same for default and non-default values.

### Steps to reproduce
_How one can reproduce the issue?_

As per the screenshot below:

```bash
# add file bundle
meltano add files airflow

# configure file bundle to disable updates
meltano config --plugin-type files airflow set _update "orchestrate/dags/meltano.py" false

# "Current value is still: True (from the default)
# This is the incorrect behavior - env-level plugin settings should take precedence over setting defaults.

# check meltano.yml
cat meltano.yml

# configure file bundle to re-enable updates (using `set`)
meltano config --plugin-type files airflow set _update "orchestrate/dags/meltano.py" true

# check meltano.yml again
cat meltano.yml

# key `orchestrate/dags/meltano.py` remains set to `false` in default environment. 
```

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

![Screenshot_2022-04-07_at_16.37.42](/uploads/85022ffbcb2b65aaeefafe8ea6ff900b/Screenshot_2022-04-07_at_16.37.42.png)

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Add guard statements and feature flags for 2.0 release

In planning for #3292+, the total amount of work for deprecations will be significant and will touch many files. To minimize merge conflicts, we can first create guard statements to raise exceptions if any deprecated code paths are touched. Combined with feature flags and/or a version check statement, we can smartly turn off _and_ turn on features in 2.0 as appropriate.

We can optionally have a `MELTANO_EXPERIMENTAL` environment variable that selectively allows new 2.0 features to be tested from pre-2.0 releases.

A large benefit of this approach is that we'd have reduced git branching, and reduced cost of resolving/managing merge conflicts.

Story points: 4.0
Active Environment Name As Env Variable

## Proposal

Would expose `MELTANO_ENVIRONMENT` to child processes, even if environment is set via another method (`--environment` at the CLI or `default_environment` in `meltano.yml`.)

## Background

Based on this thread in slack https://meltano.slack.com/archives/C01TCRBBJD7/p1643919224929049

My original comment was:

> I'm looking for a way to detect the active environment using environment variables and I know you can set a meltano environment using --environment=prod or through a MELTANO_ENVIRONMENT. Its easy to solve if I use the second method but if I define it using the --environment argument there doesnt seem to be any environment variable set. Is there a way to consistently do this?

We also now have default environments so the environment can be set in the meltano.yml too.

For any down stream process that wants to call meltano commands like I do in the dbt airflow dag generator https://gitlab.com/pnadolny13/files-airflow-dbt/-/blob/master/bundle/orchestrate/dags/generator_cache_builder.py#L36 it would be helpful to have the meltano environment exposed in the environment variables context. If I call `meltano --environment=prod invoke airflow webserver` then the dag generator should be able to access `MELTANO_ENVIRONMENT`.

It might be an anti pattern but it gives more flexibility to users so we can customize more.

Story points: 4.0
Helpful --debug tips not listed

```
visch@visch-ubuntu:~/git/tap-googleads$ pipx upgrade meltano
meltano is already at latest version 1.98.1 (location: /home/visch/.local/pipx/venvs/meltano)
```

```visch@visch-ubuntu:~/git/tap-googleads$ meltano invoke tap-googleads
Catalog discovery failed: command ['/home/visch/git/tap-googleads/tap-googleads.sh', '--config', '/home/visch/git/tap-googleads/.meltano/run/tap-googleads/tap.c180fcd0-2467-4c37-bbfb-eb5e3a8b0ef7.config.json', '--discover'] returned 1
```

The cmd line response here should have the helpful `log-level=debug` tip, not sure why it doesn't for this case.

Story points: 1.0
Improve Makefile behavior

Our Makefile doesn't have a help option AND defaults to executing a build if invoked without arguments. Makefile help/docs can be generated if you run `make explain_makefile` but you have to know that exists (or read the Makefile, in which case you no longer need the explainer). 
With some very minor changes (essentially this bit from https://gitlab.com/pandemicsyn/llamalake/-/blob/main/Makefile#L7 + a few inline comments), we could provide a help option, and use that as the default target:

```
(melty-3.8) ➜ make
api                            build api
base_image                     builds meltano/base
build                          build the ui and api
bundle                         clean up static assets and rerun UI build
clean                          clean statically generated assets
docker_images                  build docker base_image and prod_image
docker_sdist
docs_image
explain_makefile               launch a web server further explaining this makefile.
freeze_db
lint                           run python and eslint tasks
lint_eslint                    run eslint
lint_python                    run json/yaml validation and execute tox to perform python lint tasks
lock                           run poetry lock
prod_image                     builds meltano/meltano the AIO production image that includes static UI artifacts
release                        perform a release, should only be performed as part of the CI pipeline
show_lint                      show python and eslint ralated commands
test                           run tests
ui                             build the UI
```

And explicitly support `help` arg via

```

(melty-3.8) ➜ make help
api                            build api
base_image                     builds meltano/base
build                          build the ui and api
bundle                         clean up static assets and rerun UI build
clean                          clean statically generated assets
docker_images                  build docker base_image and prod_image
docker_sdist
docs_image
explain_makefile               launch a web server further explaining this makefile.
freeze_db
lint                           run python and eslint tasks
lint_eslint                    run eslint
lint_python                    run json/yaml validation and execute tox to perform python lint tasks
lock                           run poetry lock
prod_image                     builds meltano/meltano the AIO production image that includes static UI artifacts
release                        perform a release, should only be performed as part of the CI pipeline
show_lint                      show python and eslint ralated commands
test                           run tests
ui                             build the UI
```

Story points: 1.0
Executable for `meltano-map-transformer` should be `meltano-map-transform`

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?

Meltano is trying to call the wrong executable, `meltano-map-transformer`.

### What is the expected *correct* behavior?

Meltano should call `meltano-map-transform`, even though that's not the plugin name.

See in the repo: https://github.com/MeltanoLabs/meltano-map-transform/blob/134642804db3da800ed0cdc5fe9fee63584542ad/pyproject.toml#L61

### Steps to reproduce

1. try running the `meltano-map-transform` with any tap and target

### Relevant logs and/or screenshots

```
meltano --log-level debug run tap-mongodb filter-stream target-postgres

Run invocation could not be completed as block failed: Cannot start plugin meltano-map-transformer: Executable 'meltano-map-transformer' could not be found. Mapper 'meltano-map-transformer' may not havebeen installed yet using `meltano install mapper meltano-map-transformer`, or the executable name may be incorrect.
```

### Possible fixes

Change the `executable` in `discovery.yml`.


### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 1.0
Consider removing `asynctest.CoroutineMock` in favor of `mock.AsyncMock` in our tests

## Why?

- The last release of [`asynctest`](https://pypi.org/project/asynctest/) was on May 2019
- Since Python 3.8, [`AsyncMock`](https://docs.python.org/3/library/unittest.mock.html#unittest.mock.AsyncMock) is shipped with `unittest.mock`.
- For compatibility with Python 3.7, we can still use the third-party [`mock`](https://github.com/testing-cabal/mock/blob/master/CHANGELOG.rst) library.

## Considerations

- `CoroutineMock` is used in all of the following files

   - [tests/meltano/core/test_plugins_test_service.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/test_plugins_test_service.py)
   - [tests/meltano/core/runner/test_runner.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/runner/test_runner.py)
   - [tests/meltano/core/plugin/test_airflow.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/plugin/test_airflow.py)
   - [tests/meltano/core/plugin/singer/test_tap.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/plugin/singer/test_tap.py)
   - [tests/meltano/core/block/test_extract_load.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/block/test_extract_load.py)
   - [tests/meltano/core/block/test_singer.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/core/block/test_singer.py)
   - [tests/meltano/cli/test_invoke.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/cli/test_invoke.py)
   - [tests/meltano/cli/test_select.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/cli/test_select.py)
   - [tests/meltano/cli/test_run.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/cli/test_run.py)
   - [tests/meltano/cli/test_elt.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/cli/test_elt.py)
   - [tests/meltano/cli/test_config.py](https://gitlab.com/meltano/meltano/-/blob/master/tests/meltano/cli/test_config.py)

   which will have to be prettified in order for our linter checks to pass.

Story points: 4.0
When installing a custom extractor / loader, git repository URL points to Meltano gitlab namespace

This is a confusing user experience because users may try to install from a URL that does not exist. https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/cli/utils.py#L119 is where this is getting printed. As a short-term fix we should make this much more generic.

![Screen_Shot_2022-03-09_at_10.51.35_AM](/uploads/579f2f6eb1e74b2d519f5fdb0fc23733/Screen_Shot_2022-03-09_at_10.51.35_AM.png)

Story points: 1.0
Consider displaying test duration information in our CI

Some unit tests in our CI jobs seem to take significantly longer than the rest to complete, so it'd be useful to identify them.

This should be as easy as adding the `--durations=10 --durations-min=1.0` flags.

Ref: https://docs.pytest.org/en/6.2.x/usage.html?highlight=duration#profiling-test-execution-duration

Story points: 2.0
Consider using a `https` scheme in default Snowplow collector endpoint

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

Default Snowplow event emitter is created for an `http` endpoint when it should be `https`.

### What is the expected *correct* behavior?
_What should be happening?_

The default value of `snowplow.collector_endpoints` should contain an `https` endpoint.

Note that `http` may be OK if a redirect is added on the SnowcatCloud side.

### Steps to reproduce
_How one can reproduce the issue?_

NA

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

NA

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_

Change the value in `src/meltano/core/bundle/settings.yml`

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 1.0
Limit the Snowplow tracker logs

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

The snowplow tracker library is printing logs on every CLI invocation that is tracked. This is a bit distracting and not something we want users to see all the time.

### What is the expected *correct* behavior?
_What should be happening?_

Commands should not display tracking logs by default.

### Steps to reproduce
_How one can reproduce the issue?_

1. Run any CLI command that is currently tracked in GA
1. Don't use any custom `logging.yaml` so the the default logging config is used
1. Notice the log lines at the end re Snowplow tracking coming from the libray

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

![image](/uploads/30ff0d66caaccec6287ebcaf3828fdee/image.png)

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_

Set the level of the `snowplow_tracker.emitters` logger in the default config to `ERROR`.

With that change, these lines can probably be removed:

- https://gitlab.com/meltano/meltano/-/blob/cli-init-experience/src/meltano/cli/initialize.py#L36 (see https://gitlab.com/meltano/meltano/-/merge_requests/2562)
- https://gitlab.com/meltano/meltano/-/blob/02a512b62ac5a8c63a0aff25763cde915fdeec12/tests/meltano/core/test_tracking.py#L44-45

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 2.0
Plugin config in Environments requires a `config` value or `setting` definition in top level plugin declaration

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

From @pnadolny13 in https://gitlab.com/meltano/meltano/-/issues/3299#note_860273439

> So it seems like current behavior is that you cant introduce configs at the environment level unless they were already defined in the top level config or theres an explicit setting definition.

### What is the expected *correct* behavior?
_What should be happening?_

Config values defined _only_ in an Environment should work without requiring the same config to have a value or a `setting` to exist in the top-level plugin definition.

### Steps to reproduce
_How one can reproduce the issue?_

Add a config to a plugin _only_ in an Environment:

```shell
meltano --environment=dev config my-plugin set "my_config" "value"
```

Meltano fails to recognize and expose this config in the runtime plugin environment.

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

```shell
# This fails because 'SQLFLUFF_SF_USER' is referenced in my dbt but its not set
meltano invoke sqlfluff lint

# This fails because 'SQLFLUFF_SF_USER' is referenced in my dbt but its not set
meltano --environment=dev invoke sqlfluff lint

# This fails because 'SQLFLUFF_SF_USER' is referenced in my dbt but its not set
meltano --environment=prod invoke sqlfluff lint
```

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team


### Work Around Solutions

Either add an explicit setting to your plugin as shown in https://gitlab.com/meltano/meltano/-/issues/3299#note_859299619.

Or add the config key to the top level (not in an environment) plugin configuration, then Meltano recognizes it and reads the environment specific config properly. As described in https://gitlab.com/meltano/meltano/-/issues/3299#note_860273439.

Story points: 8.0
Bug - API Install Error

I noticed in this slack thread https://meltano.slack.com/archives/C01TCRBBJD7/p1646191929223089 that looks like they had an issue installing via the API due to a typo https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/api/controllers/plugins.py#L124.

Story points: 4.0
Adapter specific versions of Postgres, Redshift, and Bigquery

Follows from #3105+ 


This issue is planned as a follow-up to the initial `dbt-snowflake` release.

To have parity, we likely need these three:

- `dbt-postgres`
- `dbt-redshift`
- `dbt-bigquery`

Possible:

- `dbt-spark[PyHive]
- `dbt-presto`

Full list of dbt Labs supported adapters https://docs.getdbt.com/docs/available-adapters

Story points: 8.0
Formally remove support for "Dashboards", "Explore", and "Models"

These three relate to BI-focused features where we (Meltano) are now suggesting open source 3rd party solutions. We can therefor deprecate these features and move faster towards BI/dashboard interop with less user confusion:

- Remove support "Models", which are BI metadata models in yaml.
- Remove support "Dashboards", which are Meltano-rendered BI reports.
- Remove support for "Explore", which is an interactive analysis/query experience.

Promoted alternatives:

- Lightdash (#2786) has a Yaml-based metadata model similar to our "Models" approach, but inline with dbt project definitions.
- Superset (#2605) is a popular open source BI platform, which rich dashboard support.
- mProve (#3224) is an open source reporting tool that is entirely text-based, meaning it works plays with DevOps, Git, CI/CD, and PR review processes.

Phased deprecation:

- In a prior Meltano release, we toggled the visibility of Dashboard/Explore features in Meltano UI, setting them to 'off' by default.

Story points: 8.0