giseldo/neodataset huggingface dataset:
Dumping state as part of an elt call should pull directly from DB

In https://meltano.slack.com/archives/C01PKLU5D1R/p1636061054200300 while troubleshooting a state issue related to https://gitlab.com/meltano/meltano/-/issues/3045 folks ran into issues with the `--dump=state` command failing. This was a bit of a red herring or more a side effect - with the command simply complaining that the state file couldn't be read. When in reality the state capability wasn't being advertised by the plugin and so no state file was even being produced. 

While investigating that we realized the `meltano elt --dump=state` wasn't actually hitting the system db directly. It indirectly makes a bare `_invoke()` call using [plugin_invoker.dump](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin_invoker.py#L288), which behind the scenes triggers the [look_up_state_hook](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L165) - which in turn tries to read the state from the db, and if successful then [writes out the state to the state file](https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/tap.py#L246). After `_invoke()` completes, dump just tries to the read state file.

1. If we can deprecate/phase out the use of a state file, we could just update have the dump pull the state straight from the system db and go ahead and simplify the code in the tap ditching the state file generation code. 
2. If we need to continue generating the state file - dump could still do that - but we'd definitely want to refactor dump and look_up_state_hook to share some code. 

With either path, a single consolidated look_up_state method for both the tap and dump to use is probably the move.

Story points: 4.0
Meltano should print a warning if any of these are missing from capabilities: (`state`, `discover`, (`catalog` XOR `properties`))

There are enough instances of expensive debug operations at this point (most recently [today](https://meltano.slack.com/archives/CKHP6G5V4/p1636419311194200?thread_ts=1636399782.175500&cid=CKHP6G5V4)), that I think we should be printing an explicit warning message whenever any of these capabilities are not found in `discovery.yml`:

1. If `state` capability is not declared, then during `meltano elt`:
    > Warning: Capability 'state' is not declared for `{tap-name}`. Incremental sync bookmarks will be disabled as a result.
2. If `discover` capability is not declared, then during `meltano elt` and `meltano invoke`:
    > Warning: Capability 'discover' is not declared for `{tap-name}`. Proceeding without catalog discovery.
3. If neither `catalog` nor `properties` capability is declared, then during `meltano invoke` and `meltano etl`:
    > Warning: Capabilities `catalog` and `properties` are both undeclared for `{tap-name}`. Catalog and selection features are disabled.
4. During `add --custom`, set default capabilities to `discover,state,catalog` rather than `[]`.
5. Warn if `capabilities` is fully empty, during `meltano invoke` and perhaps also during `meltano install`.
6. `--dump` commands should fail immediately with a clear message about missing declared capabilities.
    1. `--dump=catalog` should fail hard for missing `discover`, should warn for missing `catalog`.
    1. `--dump=state` should fail hard for missing `state`.


I think these should probably print even when `--log-level=debug` is not set.

Story points: 4.0
One-Step Publish for Meltano and SDK

We have a one-click prerelease publish in the SDK. That's a simpler problem because we don't have release notes and the version number is just an auto-generated `dev` bump using the CI build number.
 
Opening this issue to figure out how we get to similar simplicity for normal deployments.

## Goals

1. Continue frequent releases.
2. Reduce cost to dev productivity.
3. Make dual-channel releases (`edge`/`main`) more viable, as proposed in https://gitlab.com/meltano/meta/-/issues/135

## Statu Quo

Here are the current manual deployments steps (excluding marketing/slack/tweets, etc.):

1. Open a release branch. (Triggers an auto-commit with the version bump.)
2. Open a tag from the release branch. (Triggers the PyPi publish.)
3. Groom the changelog. (Can be performed before, after, or in parallel with the above - since the changelog itself is not shipped.)

## Ideal State

### Changelog automation

I don't know as much about this space but I believe there are some available options.

### One-Step version bump, validation, and publish

Rather than requiring the tag to be created explicitly in a separate step, we can modify the CI pipeline to _also_ run the publish step in the special CI job that runs when a new branch is created like `release/*`.

The special CI pipeline for `release/*` tags can be removed in favor of the branch pipeline, to eliminate the need for a second step, and avoid "chaining" of auto-commits and CI jobs. We could still create the tag as part of the CI pipeline, rather than as a manual step.

### Proposed flow

Releases:

1. As release manager, I create a correctly-named release branch which triggers the release pipeline.
2. I click "Merge when pipeline succeeds" immediately - meaning, a successful CI run (including publish steps) will close out the version bump.

Change Log Grooming:

1. If needed, I create and merge a _separate_ changelog grooming MR.

Story points: 8.0
Block interface tests and mvp implementation

Issue to discuss and track our first block implementation. 


**Draft/placeholder - actively editing ~Florian**

----

#### Mergeable/deliverables:

- [x] ExtractLoadBlocks
- [x] PluginCommandBlock
- [x] IOBlock - which is parent of:
  - [ ] PublisherBlock
  - [ ] ConsumerBlock
  - [ ] TranslatorBlock
- [x] sample unit tests demonstrating construciton of and use of the ExtractLoadBlocks

Story points: 12.0
Support a Python-based Plugin Architecture

Currently all Python-specific plugin handlers live in our main `Meltano` repo and is installed in a single go.

To make a leaner install, we could (for instance) refactor "Singer EL", "dbt Transform", and "Airflow Orchestrator" into separate repositories.

For each type of plugin we want to support, we'd have to create an interface layer and decide which hooks/notifications/invocations are needed by the respective type of plugin.

Note: The refactoring effort for the Singer EL would likely be very large. It might make more sense to use orchestrators (Dagster or Airflow) first.

## How is this different?

Today, if we want to support Dagster in the same way we support Airflow, we have three levers we can control:

1. CLI-based 'command' definitions. This is basically just aliasing existing CLI commands from the tool. Not capable of deep integrations.
2. Files-based plugins. This is the least robust: we download some pre-configured files into the repo.
3. Python code-based approach. This is the only "deep" integration option we have today. This gives a lot of power to wrap the tool's capabilities, but it also requires expanding the code base of `Meltano` core repo.

This proposal would give an alternate path to the third option, enabling us to write an external plugin definition in python, and import it into the project without having it defined tightly as part of that repo.

Story points: 40.0
Prettier exception prints

On the structlog site, there's some example screenshots of exceptions that are inherently very easy to read due to formatting and structure. 

Should we look at adopting something like [rich](https://rich.readthedocs.io/) or [better-exceptions](https://github.com/Qix-/better-exceptions) for Meltano and/or the SDK?

The docs for structlog seem to imply we only need to install the libraries, but I expect it may be slightly more work, and there could be reasons I'm not thinking of why we would not want to use them.

https://www.structlog.org/en/stable/development.html:

![image](/uploads/3e18f622a003275ef6358c18dab5977c/image.png)

Story points: 2.0
Add CLI support for environments

In !2355 we did not add support for managing environments with the command line. We should have them! Based on discussion in https://gitlab.com/meltano/meltano/-/merge_requests/2383#note_713553913

At a minimum:

<details><summary>Old version</summary>

* `meltano add <environment>`
* `meltano remove <environment>`
* `meltano config <environment> (list/set/unset/reset)`
</details>

```console
$ meltano environment --help
Usage: meltano environment [OPTIONS] COMMAND [ARGS]...

Options:
  --database-uri TEXT  System database URI
  --help               Show this message and exit.

Commands:
  add    Add a new environment
  list   Enumerate available environments
  remove Remove an environment definition
```

Story points: 8.0
Add Snowplow Tracking to Meltano

https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/tracking/ga_tracker.py seems to be the primary place where GA Events are defined. All instances of `GoogleAnalyticsTracker` would have to be updated as well.

The aim of this issue would be to have 100% parity for Snowplow events with what we had for GA.

2022-02-24 Update

After syncing with Snowcat Cloud and the team we're aiming to do the following:

* Get the end-to-end data flowing to both GA and Snowplow
* This means using the structured event format to achieve parity with what we have in GA
  * This is a known sub-optimal approach as we aim to overhaul all events with proper schemas and structure
* New events should only be added following improved schema workflow and control

## Implementation Outline

<details><summary>(larger scope)</summary>

- [ ] create a new `MeltanoTracking` class to track usage, with support for many `BaseTracker` instances
- [ ] create a new `BaseTracker` class to host generic tracking methods
- [ ] create new `SnowplowTracker` class, inheriting from `BaseTracker`, to forward to Snowplow
- [ ] adapt `GoogleAnalyticsTracker` to inherit from `BaseTracker`
- [ ] replace all uses of `GoogleAnalyticsTracker` with `MeltanoTracking`

</details>

Story points: 8.0
Stand-up full Snowplow Infrastructure

Based on https://gitlab.com/meltano/meltano/-/issues/2972, we have decided to use and implement Snowplow for the open source event tracking. 

Given that the team has experience with AWS and Snowplow, deploying the infrastructure on our own AWS infra is ideal sicne it gives us full control of the data.

This issue tracks the work to get everything stood up on the AWS side. At the end of it, we should have a URL we can send events to which will be enriched and landed in S3. [This project](https://gitlab.com/meltano/saved-from-gitlab/gitlab-com-infrastructure/-/tree/master/environments/aws-snowplow) can serve as a detailed guide on how to get it working.

https://gitlab.com/gitlab-com/gitlab-com-infrastructure/-/tree/master/environments/aws-snowplow is also available!

Story points: 12.0
Add role and batch_wait_limit_seconds setting to default target-snowflake

The Wise variant is missing the optional role setting https://github.com/transferwise/pipelinewise-target-snowflake#configuration-settings

batch_wait_limit_seconds
    Integer(Default: None)
    Maximum time to wait for batch to reach batch_size_rows.

@pnadolny13 we should just do a full review to make sure everything is captured.

Story points: 1.0
Always print `--log-level=debug` guidance after failure

In slack we are often providing the guidance that people can rerun their pipelines with `--log-level=debug` - generally needed when a tap or target cannot properly configure itself or fails auth.

What do folks think about _always_ printing this after failure?

Something like:

> .... has failed. For more detailed log messages check '...' log directory or rerun with the --log-level=debug CLI flag.

Related slack thread: https://meltano.slack.com/archives/C013EKWA2Q1/p1634836288002100?thread_ts=1634832556.001800&cid=C013EKWA2Q1

Story points: 4.0
Uninstall documentation

### Problem to solve
Need to make it clearer that the way to force a reinstall of a tap is to delete the directory in the `.meltano` directory.

Currently there is documentation on how to [install](https://meltano.com/docs/plugin-management.html#custom-plugins) a custom plugin, but especially when using custom taps, it isn't immediately clear how to remove them. 


### Target audience
Developers building custom taps and using meltano as part of the development process.


### Further details
This currently is relatively undocumented, only in [Slack](https://meltano.slack.com/archives/CFG3C3C66/p1594084482487700?thread_ts=1593999699.428400&cid=CFG3C3C66) is it made clear that removing files in `.meltano/extractors` will trigger a complete reinstall, ie once deleted, `meltano install <type> <name> --reinstall` will reinstall it.



### Proposal
Provide a proper reinstall option, ie `meltano install <type> <name> --reinstall`, with the same impact of a `rm -rf .meltano/extractors/tap-<name>

Currently in the [docs](https://meltano.com/docs/plugin-management.html#installing-your-project-s-plugins) it indicates that reinstall is possible, but this does not do a complete reinstall.

> To (re)install a specific plugin in your project, use meltano install <type> <name>, e.g. meltano install extractor tap-gitlab.


### What does success look like, and how can we measure that?
Able to remove a tap without deleting files in `.meltano`


### Links / references

* [Install docs](https://meltano.com/docs/plugin-management.html#installing-your-project-s-plugins)
* [Slack conversation](https://meltano.slack.com/archives/CFG3C3C66/p1594084482487700?thread_ts=1593999699.428400&cid=CFG3C3C66)

_Please note that this was taken from GitLab, to be changed accordingly_

Story points: 1.0
meltano | Transformation failed (2): Env var required but not provided: 'MELTANO_LOAD_SCHEMA'

According to @DouweM, this can occur if a custom loader that doesn’t define a schema setting, so the env var doesn’t get populated and dbt doesn’t know what schema to read from.

I think Meltano is probably functioning as designed this feels like a bug to me. In a world where we have 10s of taps loading to the same environment (#2869), dbt should probably be mapping to schema or tap names statically (in yaml config) rather than dynamically (in env vars).

We should also strive to improve the user experience for users creating their own transformation sets, which will increasingly become the norm as we are moving away from the more fragile `transform` packages that expect specific database dialect (generally postgres) and a specific variant of the given tap.

Labeling as a bug because of the user impact, although in fairness, I do think this is functioning as originally designed.

Story points: 8.0
Problems with environment variables taking precedence over declared ones

If a plugin configuration option is set to a specific environment variable, but that config option as a default environment variable it can be configured with, and _both_ are defined, Meltano uses the default one rather than the explicitly declared one.

Example:

```bash
meltano init demo
meltano add extractor tap-gitlab
meltano config tap-gitlab set private_token '$MY_REAL_GITLAB_API_TOKEN'
export GITLAB_API_TOKEN=wrong-token
export MY_REAL_GITLAB_API_TOKEN=right-token
meltano config tap-gitlab
```

```
{
  "api_url": "https://gitlab.com",
  "private_token": "wrong-token",
  "groups": "",
  "projects": "",
  "ultimate_license": false,
  "fetch_merge_request_commits": false,
  "fetch_pipelines_extended": false
}
```

To me, this is unexpected behavior. I would expect `GITLAB_API_TOKEN` to be used if I hadn't specified anything for `api_url`, but since I explicitly told Meltano to use `MY_REAL_GITLAB_API_TOKEN` I'd expect that to take precedence.

## Proposed Solution (AJ)

I think we should deprecate `env: CUSTOM_ENV_NAME` and `env_aliases: [...]` in favor of the more explicit `<plugin_name>_<setting_name>`. I've created #3120 to track this.

Story points: 4.0
Hide `Explore` and `Dashboards` in UI by default

To help new users, I think we should remove `Explore` and `Dashboards` from the new user experience here:

![image](/uploads/38ae54b0e73c218d21120cdceb4a6474/image.png)

Rather than have this on by default, I would like users to have to provide an environment variable to enable them, with the caveat that this is legacy functionality and the corresponding activities are not being actively supported.

This is a half-step between removing the functionality entirely and would focus on reducing frustration for new users who naturally and understandably gravitate to the UI experience first.

Presumably, users could optionally type in the URLs directly if needed. Goal here is to reduce discovery during new-user experience, not so much prevent someone who already is using this functionality from leveraging it.

Story points: 1.0
Revert per build network bridge work around

As a workaround for https://gitlab.com/meltano/meltano/-/issues/2951 where we were seeing network timeouts on package installs [we deployed a workaround](https://gitlab.com/meltano/meltano/-/commit/4f2ce5b060a45aff37a580710f3b48c1c7215f63) recommended upstream by Gitlab. However, Gitlab has since worked with their provider to resolve the underlying issue.

The workaround we deployed is a subtle change in behavior in how networking in our CI containers is built/configured so we should revert this change until we want to (intentionally) make use of this feature.

Story points: 1.0
API endpoint for testing an extractor instantiates an ABC

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_

From a slack converstation (https://meltano.slack.com/archives/CMN8HELB0/p1632924967171100), it seems like the test endpoint is instantiating the abstract base class `PluginTestService` instead of `ExtractorTestService`.

### What is the expected *correct* behavior?
_What should be happening?_

The endpoint should test an extractor correctly.

### Steps to reproduce
_How one can reproduce the issue?_

Try hitting the `POST /<plugin_ref:plugin_ref>/configuration/test` endpoint.

### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_

```
[2021-09-29 16:13:56,137] [74486|MainThread|meltano.api.app] [ERROR] Exception on /api/v1/orchestrations/extractors/tap-hubspot/configuration/test [POST]
[...]
TypeError: Can't instantiate abstract class PluginTestService with abstract method validate
[2021-09-29 16:13:56,140] [74486|MainThread|meltano.api] [INFO] Error: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
```

### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_

Replace `PluginTestService` with `ExtractorTestService` in the function body.

### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 1.0
Make Docker images available in AWS public ECR gallery

Using Docker has become pretty annoying if you use a code pipeline. You invariably will get "toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading:".

Since we build stuff with Meltano, we get this quite a bit. Is it possible for Meltano to upload their images to the AWS ECR public registry please as well? See here: https://gallery.ecr.aws/

Story points: 4.0
Create a GitHub Action for Meltano ELT

## Summary
[//]: # (Concisely summarize the feature you are proposing.)

GitHub Actions are a CI/CD offering with capabilities similar to those of GitLab CI, except that they can be published to the [a marketplace](https://docs.github.com/en/actions/creating-actions/publishing-actions-in-github-marketplace) where they get more visibility and users can add them to their projects with one click.

This action would be useful for developers that are first trying out Meltano, or don't need a full Kubernetes/cloud deployment with an orchestrator, metrics UIs, etc.

## Proposed benefits
[//]: # (Concisely summarize the benefits this feature would bring to yourself and other users.)

Give users that want to try out Meltano for ELT a way that is quick and easy.

## Proposal details
[//]: # (In as much detail as you are able, describe the feature you'd like to build or would like to see built.)

- A [composite action](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action) is probably easier but we can consider a [docker action](https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action) too.
  Either way, the action would install the latest or some configurable Meltano version for the user and any other dependencies, and also run `meltano install`.
- Inputs to the job could be: `tap`, `target`, `job_id`, `transform` and `log_level` to compose a command like

  ```shell
  meltano --log-level={log_level} elt {tap} {target} --job_id={job_id} --transform={transform}
  ```

## Best reasons not to build
[//]: # (Will this negatively affect any existing functionality? Do you anticipate any breaking changes versus what may already be working today? Make the counter-argument to your proposal here.)

This only solves ELT and very basic orchestration needs. No fancy UI for jobs and retries are manual.

Story points: 1.0
Create Local Deployment File Bundle

Having produced a proof of concept local deployment (#2910) we would now like to publish this as a [File Bundle](https://meltano.com/docs/plugins.html#file-bundles) for installation via `meltano add`. This includes:

- Publishing the Helm Charts for Meltano and Airflow
  - Helm expects charts to be hosted under plain http(s) urls. I intend to use GitLab Pages to fulfil this requirement.
- Publishing the Terraform Modules for Kind, Postgres and Docker
  - Terraform supports installing modules [directly from a git repository](https://www.terraform.io/docs/language/modules/sources.html#generic-git-repository)
- Creating a File Bundle that includes an `infrastructure/local` dir, `main.tf` file and other artefacts needed to launch the local deployment.

Story points: 12.0