giseldo/neodataset huggingface dataset:
Support Deployment to AWS Lambda (Serverless)

AWS Lambda provides a fully-managed (serverless) environment that includes both scheduling and compute (including docker). By supporting lambda as a deployment option and orchestrator for Meltano we can provide an easy onramp into the Meltano ecosystem for customers already on AWS but otherwise unwilling to host and manage a self-deployed orchestrator like Airflow.

Integration would include:

- A lambda-compatible docker container for Meltano
- A way to manage state utilising only serverless services (DynamoDB, Aurora Serverless, S3 etc.)
- A way to gracefully handle the 15min max execution time of Lambda.

There are several possible approaches to the final requirement:

- Include a new `--max-execution-time` flag that terminates the tap gracefully (emitting state) even when more upstream records are available. This allows meltano to sync data using successive Lambda invocations without being 'reset' to a previous state due to timeouts, but carries the risk that streams fall behind and never catch up with the upstream source.
- Parallelise pipelines using 'one stream per function' to minimise the risk of several streams in a single pipeline approaching the limit.

Story points: 40.0
Support Deployment Plugins

For Meltano to truly be a 'one-stop-shop' for your data platform, it needs to be able to deploy the individual tools and services that make up that platform. To achieve this Meltano should support deployment plugins, being existing tools for deploying and managing running infrastructure (such as Helm, Terraform, Ansible, Salt, Cloudformation). For those just getting started, Meltano should aim to provide opinionated base config for common scenarios (e.g. containers on Kubernetes), and for power users of these deployment tools the config and internals should still be accessible and easily customisable via existing Meltano config routes.

As a rough guide, the CLI could look something like:

```bash
# First a user would add a deployment. This would install all the necessary tooling to actually deploy to this platform during `meltano install`. Meltano will use all of the context of installed plugins, orchestrators and config to create appropriate deployment artefacts.
meltano add deployment --platform kubernetes --environment prod

# Users can optionally configure their deployment
meltano config deployment --environment prod set namespace "team-data"

# Finally a new deployment can be executed 🚀
meltano deploy --environment prod

# Individual tools can also be invoked and used directly
meltano invoke helm list --all-namespaces --all
```

Story points: 40.0
Refresh catalog on every invoke (fresh_catalog: true))

I want to have Meltano build a new catalog every run when running `meltano invoke tap-oracle`

A key at the SingerPlugin level probably makes sense, maybe call it `fresh_catalog` default to false? 

More specific than #2850 as #2627 didn't solve what I'm after. What I really want is a way to manage and watch my catalog change over time (#2677 / #2805 ), but this issue will be an incremental improvement over where I"m at today. 

Today I delete the catalog and cache key from `.meltano/run/tap-name/*`

Story points: 4.0
Refactor to convert plugins to leverage async patterns in hooks

As a follow up to https://gitlab.com/meltano/meltano/-/issues/2868 (refactor SingerTarget to own bookmark writer) and a next step to https://gitlab.com/meltano/meltano/-/issues/2725 (Refactor `meltano elt`, `SingerRunner` and `DbtRunner` to be composable). We should refactor the plugins various before/after hooks and supporting methods to be async.

Story points: 8.0
A file bundle is added twice via `--include-related` if its configuration does not contain the `update` field

### What is the current *bug* behavior?

When adding a plugin with `--include-related`, Meltano adds a related file bundle plugin twice if it does not include the `update` field in its schema. The file bundles installs correctly if `update` is given with any key-value pair.


### What is the expected *correct* behavior?

The file bundle should be added only once with `--include-related`, regardless of whether or not the `update` field is provided.

### Steps to reproduce

1. Define a custom file bundle plugin (e.g. `files-google-analytics`) sharing a namespace of some other plugin (e.g. `tap-google-analytics` with namespace `tap_google_analytics`) **WITHOUT** any `update` field

```yaml
files:
  name: files-google-analytics
  namespace: tap_google_analytics
  repo: https://github.com/me/files-google-analytics
  pip_url: https://github.com/me/files-google-analytics
```

1. Add the file bundle via `--include-related` (e.g. `meltano add extractor tap-google-analytics --include related`)
1. Observe file bundle (`files-google-analytics`) add twice, with two occurences of `Adding related file bundle 'files-google-analytics' to your Meltano project...` in the output log.

### Relevant logs and/or screenshots

*Installs correctly*

```yaml
files:
  name: files-google-analytics
  namespace: tap_google_analytics
  update:
    path/to/some/file: true
  repo: https://github.com/me/files-google-analytics
  pip_url: https://github.com/me/files-google-analytics
```

```yaml
files:
  name: files-google-analytics
  namespace: tap_google_analytics
  update:
    path/to/some/file: false
  repo: https://github.com/me/files-google-analytics
  pip_url: https://github.com/me/files-google-analytics
```

*Installs incorrectly*

```yaml
files:
  name: files-google-analytics
  namespace: tap_google_analytics
  repo: https://github.com/me/files-google-analytics
  pip_url: https://github.com/me/files-google-analytics
```

```yaml
files:
  name: files-google-analytics
  namespace: tap_google_analytics
  update: {}
  repo: https://github.com/me/files-google-analytics
  pip_url: https://github.com/me/files-google-analytics
```

### Possible fixes

If `update` is not provided, Meltano [defaults it to an empty object (`{}`)](https://meltano.com/docs/plugins.html#update-extra), which I assume is the root cause of this bug. Specifying `update: {}` manually yields the same results.

### Further regression test

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Additional Metadata for PyPI

## Summary

Add more project metadata fields for PyPI.

## Proposed benefits
[//]: # (Concisely summarize the benefits this feature would bring to yourself and other users.)

- More resource links in our PyPI page: https://pypi.org/project/meltano/
- Better SourceRank score: https://libraries.io/pypi/meltano/sourcerank

## Proposal details
[//]: # (In as much detail as you are able, describe the feature you'd like to build or would like to see built.)

[According to the Poetry docs](https://python-poetry.org/docs/pyproject/), we seem to be missing a few optional metadata fields:

- `maintainers`
- `homepage` (https://meltano.com)
- `documentation` (https://meltano.com/docs)
- `keywords`

We can also [add custom URLs](https://python-poetry.org/docs/pyproject/#urls):

```toml
[tool.poetry.urls]
"Issue Tracker" = "https://gitlab.com/meltano/meltano/-/issues"
"Twitter" = "https://twitter.com/meltanodata/"
"Changelog" = "https://gitlab.com/meltano/meltano/-/blob/master/CHANGELOG.md"
"Slack" = "https://join.slack.com/t/meltano/shared_invite/zt-obgpdeba-7yrqKhwyMBfdHDXsZY8G7Q"
"GitHub Mirror" = "https://github.com/meltano/meltano"
"Youtube" = "https://www.youtube.com/meltano"
```

## Best reasons not to build
[//]: # (Will this negatively affect any existing functionality? Do you anticipate any breaking changes versus what may already be working today? Make the counter-argument to your proposal here.)

None that I can think of.

A similar issue for the SDK is https://gitlab.com/meltano/sdk/-/issues/175.

Story points: 1.0
"Yank" older meltano versions which are missing Python constraints

We have a problem right now that many users are installing meltano on Python 3.9 - which is not supported - but the install is moving forward anyway on older versions (specifically `1.67`) which were published without an explicit max supported Python version. The net result is (1) the new user gets a stale version of Meltano, and (2) they don't get any messaging about 3.9 being unsupported.

## Long-term mitigation needed

Although we are soon going to launch 3.9 support (#2545) this issue will resurface again in October when [Python 3.10 is expected to release](https://www.google.com/search?q=python+3.10+release+date).

## "Yanking" old versions

According to the PyPi docs:

![image](/uploads/89cadb432804675f08528e8b7952d389/image.png)

- https://pypi.org/help/#yanked
- https://www.python.org/dev/peps/pep-0592/

## Risk of Resurgence Post-Fix

If we do the one-time yank of old versions, this issue should not resurface because all published versions have a max supported version.

Story points: 1.0
Start tracking open source licenses

We should start tracking the licenses of the open source packages Meltano depends on, to make sure Meltano can sitll be licensed Apache 2.

We can use https://docs.gitlab.com/ee/user/compliance/license_compliance/. It uses https://github.com/pivotal/LicenseFinder, which requires `requirements.txt`, which we don't have since we're using Poetry, but we should be able to generate that on the fly using the `SETUP_CMD` env var described in https://docs.gitlab.com/ee/user/compliance/license_compliance/#installing-custom-dependencies.

Story points: 12.0
Document where domains/DNS is managed

We should document on https://meltano.com/handbook/engineering/ where our various domain names are registered (AWS, NameCheap, Gandi) and where DNS is managed (typically SiteGround).

Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12688

@aaronsteers I seem to recall you already created an issue for not, but if not, this can be it.

Story points: 2.0
Coroutine async issue with parallel installs version 1.77.0

We are installing meltano inside a Docker container with a fairly simple `pip install upgrade pip` + `pip install meltano` with the conditions:

Docker Image: Ubuntu 20.04
Python version: 3.8.5
Meltano version: 1.77.0 (but applies to 1.73.0 - 1.77.0)

and the command `meltano install` causes the following problem:

```
meltano install
Installing 3 plugins...
Installing loader 'target-mysql'...
Installing loader 'target-postgres'...
Installing extractor 'tap-s3-csv'...
'coroutine' object has no attribute 'read'
```

We are installing 1 Singer tap and 2 targets, not a particularly complicated setup. To solve this problem, we are currently using Meltano version 1.72.0. Note that I am totally new to Meltano, so I can't really propose comprehensive solutions. Please let me know if you need any more information.

UPDATE: That was the original situation but once I did `apt-get python3-venv`, all of these issues got solved and meltano version 1.77.0 was installed as it should be. So, maybe we just need better documentation on how to install Meltano in a custom docker container.

Story points: 4.0
Copy updated dbt and transform settings from discovery.yml due to dbt 0.19.1 upgrade

The following discussions from !2170 should be addressed:

- [ ] @tayloramurphy started a [discussion](https://gitlab.com/meltano/meltano/-/merge_requests/2170#note_602038447):  (+3 comments)

    > @aaronsteers will you also make an MR to update [discovery.yml](https://gitlab.com/meltano/hub/-/blob/main/_data/meltano/discovery.yml), [dbt.yml](https://gitlab.com/meltano/hub/-/blob/main/_data/meltano/transformers/dbt.yml), and the [transform yamls](https://gitlab.com/meltano/hub/-/tree/main/_data/meltano/transforms)?

Story points: 2.0
Generate a "run_results" artifact after any Meltano run

In discussion with Stephen Bailey from Immuta he brought up the idea of having some kind of run artifact similar to what dbt does with [their artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts#when-are-artifacts-produced). 

At a minimum it would be good to have some kind of summary of an `elt` run to be able to ship anywhere, but I could also see us having the equivalent of a manifest generated as well. This would tie in well with https://gitlab.com/meltano/meltano/-/issues/2611

Story points: 8.0
Update location of simplified Spec

### What is the current *bug* behavior?
Links from google and the Meltano blog point to:

https://meltano.com/docs/singer-spec.html

which returns a 404 Not Found error. Specific blog post:

https://meltano.com/blog/2021/04/07/a-simplified-take-on-the-singer-spec/

### What is the expected *correct* behavior?

New URL is:

https://hub.meltano.com/singer/spec

Story points: 1.0
Publish dbt packages to https://hub.getdbt.com/

Publishing to dbtHub could give ~~2~~ 3 significant benefits:

1. Allow installation via semantic versioning, which could give a better versioning option vs the branch-pinning solution for `config-version-2` updates needed for #2694. (Semver range restrictions are only available for hub-hosted packages.)
2. Introduce dbt users to Meltano by surfacing our work there on https://hub.getdbt.com/
3. Attract users (and community contributors) for those packages.

@tayloramurphy, @DouweM - Thoughts?

Checklist:

- [ ] Documentation:
    - [ ] Readmes should answer: "What is Meltano?" and "What is Singer?".
    - [ ] Readmes should mention all streams need to be selected.
    - [ ] Readmes should have a short walkthrough (and/or link to one) using meltano to sink from tap to target and run the transform.
    - [ ] We should note which database(s) each package has been tested on. If possible, we should try to add Snowflake as well as Postgres.
- [ ] CI Testing
    - [ ] We should require an end-to-end CI test for each before publishing to Meltano hub. (This could be the first Meltano experience for many new users, and I expect there could be a large demand/interest.)
- [ ] Migration:
    - [ ] GitHub project migration or mirroring (barring GitLab support on Meltano side https://github.com/fishtown-analytics/hub.getdbt.com/issues/571)

Story points: 8.0
Safely skip `docker build` step (`meltano_dev`)

The `meltano_dev` step takes approximately 14-15 minutes, which is between 33 and 50 percent of average CI build times. We can reduce this time with one or more of the following methods: 

1. Only rebuilding the docker image when dependencies change:
    1. Create a hash of `pyproject.toml` and `poetry.lock` and use a combination/hash of those hashes as the docker tag.
    2. Before running the docker build, check if the docker hash already exists as a tag. If so, reuse the existing image in subsequent steps (skips the build time).
    3. Create a manual build step which can be used to force docker image build a build even when nothing has changed.
2. Running 'fail-early' tests redundantly without a fully-rebuilt docker image.
    1. Start with a stock python image and/or the image from the last stable master build.
    2. Run `poetry install` - which should finish immediately if there's nothing to do.
    3. Run at minimum `pytest [3.8]` and `python_lint [3.8]` using the older image. The results should mostly be reduntent with the similar tests, except they should run (and fail) about 12 minutes faster.
    4. As with the above, if we choose to use the last master image rather than a stock python 3.8 image, we will want a manual build step option to replace whatever prior/cached image with the latest on the current branch.

Any other options/strategies I'm missing?

Story points: 8.0
Docs for ephemeral production pipelines running (incrementally) with no Postgres backend

In creating the pipeline for the Hub, I did not find docs for how to productionalize a pipeline while not requiring a postgres database. I was thinking of adding a short guide in `Tutorials` or in `Productionalization`.

Here are the related resources I found:

- How to use the `state` extra: https://meltano.com/docs/plugins.html#state-extra
- `--dump` option with example: https://meltano.com/docs/command-line-interface.html#examples

I think the process we want for the Gitlab pipeline is:

1. Load `state.json` from external storage (in our case, either S3 or a Gitlab CI artifact store.)
2. Ensure the `state` extra is configured.
3. ~~**To confirm:** during execution, `state.json` should hopefully be continually updated by meltano. (I think _we do not_ need the `--dump` option, in this case.)~~ During load, state is incremented updated in the system db (only).
   - State must be exported by running a second invocation with the `--dump=state` CLI option.
4. Catch failures and upload the new state regardless, so that incremental processing will continue on the next run even if the operation is interrupted.
5. Still raise a failure (return non-zero) if the pipeline fails, so the job status reflects the execution result.

Story points: 4.0
Add Lightdash as a utility plugin

This is a new BI tool: open source, (apparently) CI-friendly, and integrated with dbt.

https://www.lightdash.com/


License: MIT

Sample code (inline with a dbt `schema.yml` file):

![image](/uploads/5693ca56e20bfdac84bd0949f7401400/image.png)
From https://docs.lightdash.com/guides/how-to-create-metrics

Story points: 4.0
health check return 401 when MELTANO_AUTHENTICATION is set to true

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
_What is happening now?_
health check (/api/v1/health) return 401 when MELTANO_AUTHENTICATION is set to true.

### What is the expected *correct* behavior?
_What should be happening?_
it should return a 200 without being authentificated.


### Steps to reproduce
_How one can reproduce the issue?_
set MELTANO_AUTHENTICATION to true, run meltano ui and then check the response at http://host:port/api/v1/health
### Relevant logs and/or screenshots
_Please use code blocks (\`\`\`) to format console output_
i will add it later.


### Possible fixes
_If you can, link to the line of code that might be responsible for the problem or suggest a fix_



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 8.0
Minor tech debt from pre-commit, flakehell, and flake8

As noted in https://gitlab.com/meltano/meltano/-/merge_requests/2150#note_591431777, there's some minor tech debt I want to document for future review.

1. [ ] `pre-commit` hooks are not version locked with poetry
2. [ ] `flakehell` uses a private interface of `flake8`, which broke when `flake8` released `3.9.1` (also for `3.9.2` and likely future releases as well).
3. [ ] Workaround added in https://gitlab.com/meltano/meltano/-/merge_requests/2150 adds a `extended_default_ignore=[]` declaration into `pyproject.toml`, which fixes the issue.

In future, it would be good to see if we can sync the pre-commit hooks with poetry (if possible) and/or other mitigations.

Story points: 20.0
Update stale `meltano add --custom` doc example

I missed this during the release and will need to follow up. The docs are correct in how they instruct the user to proceed, but the example invocation of the `add --custom` command is stale by at least a couple releases.

Search `meltano add --custom extractor tap-covid-19` to find the sample - and update the page with the latest cli prompts and input values.

Story points: 2.0
Change default behavior in `meltano config ... set` when `kind` is missing

We should considering altering how Meltano behaves when `kind` is missing from specific settings. The default behavior when `kind` is not know would be to treat the value as insensitive. We may instead want to prompt the user to specify the `kind` and/or the sensitivity level of an otherwise undeclared setting value.

For instance, prompting a user to specify it if configuring with `meltano config <plugin> set <setting> <value>. Especially for custom taps (including many taps discovered from the hub), the `kind` and therefor the sensitivity level is not known, and we should probably not automatically default to that value _not_ being a secret. 

This could also serve as a catch for typos or accidentally-incorrect setting names: rather than going through the wizard's steps and storing under the provided setting name, the user may realize they've input an incorrect setting name and abort, then retry the process with the correct setting name.

Here's an example invocation. The result of the below (using an unknown/incorrect setting name) is that the setting gets stored in yaml as clear text:

```
% poetry run meltano config target-jsonl set myset thisisnotasecret
Loader 'target-jsonl' setting 'myset' was set in `meltano.yml`: 'thisisnotasecret'
```


## Proposal

- We should at minimum warn the user that `kind` is missing.
- If an interactive prompt is available, we can ask the user interactively what the proper `kind` value is _before_ serializing the new setting value.
  - The setting's `kind` entry would be serialized to yaml as well.
- We can add a `--quiet` flag optionally, to mute the new behavior and treat the setting as `kind: string` if not set. (Same as current status quo.)

## Proposal (Updated 2022-04-16)

- Combining with #3227+:
  - If `--interactive` is used, we can _also_ prompt for setting config (including `kind`) at the same time as prompting for the new value.
  - If `--interactive` is not set, we can fail with a prompt to use `--interactive` _or_ we can automatically switch to `--interactive` mode if `kind` is missing or the setting is not yet defined.

Story points: 4.0
Document how we're deploying Meltano for the hub

With https://gitlab.com/meltano/hub/-/issues/15 we're truly dogfooding Meltano (https://gitlab.com/meltano/meta/-/issues/4). This is a great opportunity to create some relevant content around this effort.

- [ ] Blog Post on how we're using Meltano to build the Hub
- [ ] Documentation update in several areas:
  - [ ] Cross-link in https://meltano.com/docs/production.html
  - [ ] We should have a section on the Hub on how we're building the hub "Architecture" 
  - [ ] Create a "Production Deployments in the Wild" section or something like that within the Meltano documentation. We can link ours and the GitLab data team's.

Story points: 4.0