giseldo/neodataset huggingface dataset:
Allow extractor entities to be selected in UI

We already had an "Entity selection" modal before, but it was removed in https://gitlab.com/meltano/meltano/-/merge_requests/1157:

![](https://gitlab.com/meltano/meltano/uploads/4d03378285cecd17fbb9e8e537838471/Screen_Shot_2019-08-14_at_8.30.04_AM.png)

I think we should:
- [ ] Restore the `EntitiesSelectorModal` component from that MR
- [ ] Add a new "Select entities" button to installed extractors on the Extract page, provided they support the `discovery` and `catalog`/`properties` capabilities
- [ ] Make it clear that with anything but the full set of properties, Meltano's built-in transforms may not work, as described in https://gitlab.com/meltano/meltano/-/issues/876#note_240275676

Story points: 8.0
Add "Retry" button to pipeline logs view and list when run failed

### Problem to solve
(Summarize the problem we are trying to solve in the form of we need to do [A], so that [B] can [C])

We need to *add a pipeline retry button to the Meltano UI Pipeline logs view*, so that when I see that something went wrong I can retry it easily. 

![Screen_Shot_2020-01-18_at_7.15.57_AM](/uploads/328de49638e6d3d5e4285cd0b07d5827/Screen_Shot_2020-01-18_at_7.15.57_AM.jpg)

For example, I can see that "Nothing to do" happened, but something should have happened. 
I want to be able to make changes and then just hit "retry".

### Target audience
(For whom are we doing this? Include a persona)

Anyone who is editing or building their own analyses. 

### Further details
(Include use cases, benefits, and/or goals)

It will make it easier to trouble shoot. 
This helps make solve the "Have you tried restarting it?" problem.


### Proposal
(How are we going to solve the problem? Try to include the user journey)
I propose a Retry button in the bottom right of the modal.


### What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)

### Regression test
(Ensure the feature doesn't cause any regressions)
- [ ] Write adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

### Links / references

_Please note that this was taken from GitLab, to be changed accordingly_

Story points: 4.0
Combine with ML Flow to Scale Knowledge

### Problem to solve

Data science/ML experiments are often difficult to communicate to fellow team members and executive. When communicating to team members, the experiment has to be **reproducible** in order for them to have more technical depth. When communicating with executive, the experiment has to be **communicated in a simple manner** in order to have investment from leadership. Not only do executives and team members are stake holders, but other people in the company might be too. Scaling knowledge of experiments is difficult within any company. 

### Target audience

Data Scientist

*  Frustrated at a company process designed for software engineers
*  Might able to communicate success, but unable to explain the failed experiments leading up to that success. Failed experiments consists of the core pieces of insight gained by the scientist. Stakeholders often wonder why the process take so long.
* .Might be able to communicate successful experiments to a few stakeholders, but unable to do so with all those that might be interested. Uable to scale their communication.
* Frustrated at the slow decision making when the number of stakeholders that needed to be communicated to are high
* Frustrated at the review process designed for software engineers -> Git code management system like Bitbucket does not have the necessary features to support the review process needed for high quality feedback for peers -> want to have high quality feedback on experiments to produce high quality results

 **As an organization grows, how do we make sure that an insight uncovered by one person effectively transfers beyond the targeted recipient?**

### Further details

One of the major problems is not only sharing reports and analytics but also ML experiments so that knowledge can be pervasive. There have been attempts like this such as [Airbnb Knowledge Repo](https://github.com/airbnb/knowledge-repo) but it's an underinvested area. It's still an unsolved problem within the data science community.

### Proposal
Combine it with MLflow Projects to have reproducible experiments.

### What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)

Have a solution that covers 5 main areas:
1.  Reproduciblity
2.  Quality
3.  Consumability of content
4.  Discoverability
5.  Learning/growth

Reference: [Scaling Knowledge at Airbnb](https://medium.com/airbnb-engineering/scaling-knowledge-at-airbnb-875d73eff091)

### Links / references

_Please note that this was taken from GitLab, to be changed accordingly_

https://github.com/airbnb/knowledge-repo

https://mlflow.org

https://medium.com/airbnb-engineering/scaling-knowledge-at-airbnb-875d73eff091

Story points: 1.0
Add Snowflake dialect to PyPIKA

### Problem to solve
We are currently using PyPIKA to generate the SQL queries for the Meltano UI. As we want to use Snowflake as a data warehouse, we need to make sure our SQL generation engine supports it. We'll need to add a Snowflake dialect to PyPIKA, which might involve sending back as a contribution or as a separate package. Once the dialect in, PyPIKA should be able to generate queries compatible with Snowflake specifics.

### Target audience
Data Analyst


### Further details
Without this, the generated SQL might cause an error when queried.


### Proposal

  - [ ] Investigate Snowflake SQL support
  - [ ] Investigate PyPIKA dialect handling 
  - [ ] Investigate PyPIKA dialect integration
  - [ ] Build the dialect

### What does success look like, and how can we measure that?
Our SQL generation engine can output Snowflake compatible SQL.

### Links / references

Related to #428 

https://github.com/kayak/pypika/blob/master/pypika/dialects.py

Story points: 3.0
Add Snowflake dialect the database settings

Related to #428 

### Problem to solve
We need to add a Snowflake dialect in the database settings, along with any dialect-specific configuration keys in the database settings so that Meltano API can connect to Snowflake for querying.


### Target audience
Audience: Data Analyst


### What does success look like, and how can we measure that?
We are able to add a Snowflake connection in the Meltano UI database settings.


### Links / references

_Please note that this was taken from GitLab, to be changed accordingly_

Story points: 1.0
Create a connector for Snowflake in the UI

### Problem to solve
(Summarize the problem we are trying to solve in the form of we need to do [A], so that [B] can [C])
Currently, if we want to use snowflake as a target, we can run the ETL, but a user cannot use the UI to analyze data

### Target audience
(For whom are we doing this? Include a persona)
Anyone replicating the tutorials

### Further details
(Include use cases, benefits, and/or goals)
Have a full working demo with snowflake as a target

### Proposal
(How are we going to solve the problem? Try to include the user journey)
Create a connector for Snowflake - @iroussos, let's break down the problem if needed

### What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)
AC: We can run each demo e2e with target snowflake and play with the data in the UI

### Links / references

_Please note that this was taken from GitLab, to be changed accordingly_

Story points: 4.0
Planning/Exploration Sprint - should Meltano offer a hosted sandbox experience?

### Full Opportunity Assessment

@valexieva please add link here once written: --> both links below

### Problem(s) to solve
Our current product adoption path assumes users launch Meltano in their local development environment (their computer), however due to the size of many data sets this can cause issues with memory, performance, etc. which can limit user adoption and recurring MAU

However, users also may be using Meltano on data that we do not want to responsible for hosting (PII, GDPR, etc.) so we also need to think about guiding them to use a production environment that makes sense for their needs.

### Proposed Solution (to Validate)
Offer the user a way to use Meltano while hosting their data somewhere that can handle a big data set. 

Will explore pros/cons of different options, including:
* Self-hosting Meltano on the user's existing infrastructure
* Self-hosting Meltano on a one-click install (e.g. Bitnami, Dreamhost, Amazon, etc.)
* Meltano hosted sandbox


### Tasks to Complete Planning/Exploration Sprint
* [x] Write an opportunity assessment for a hosted sandbox experience (template here: https://docs.google.com/document/d/1rKEihpSxzC_a499xzbuIh9Jd-MfteOaUTMVqQJBzAtA/edit# - go / no-go + recommendation
* [x] Write an opportunity assessment for docker (template here: https://docs.google.com/document/d/1OG2HzICuj0v2C1itznPoW4mI5KtroSDKSmDIPxet0OE/edit# - go / no-go + recommendation
* [x] Collect and discuss advantages / disadvantages of all options in the context of the product
* [ ] ~~Architectural design proposal, including development environment/ deployment to production~~
* [ ] ~~Feasibility discussion~~
* [ ] ~~Lightweight development design for a 1 week sprint experiment~~
* [x] Go / no-go for immediate action versus place in the backlog. **Decision is: NO GO**, we will execute on our Docker image for now and revisit later

Story points: 4.0
Add workflow for custom taps and targets

### Problem to solve
_Summarize the problem we are trying to solve in the form of we need to do [A], so that [B] can [C]_

We currently don't support many taps and targets and this poses risk to adoption. We would like to have a very clear explanation / workflow for anyone willing to contribute

### Target audience
(For whom are we doing this? Include a persona)

Developers

### Further details
(Include use cases, benefits, and/or goals)

By adding more taps and targets, we hope to be able to increase adoption by allowing companies to bring many different data sources in 1.
Developing taps and targets in Singer is painful and we believe that by making this process smoother, we can improve the Singer contributing community, which is currently a roadblock to adoption for us.


### Proposal
(How are we going to solve the problem? Try to include the user journey)

- [x] Add custom switch to `meltano add` - we should be able to add any tap/target bypassing discovery.yml from the CLOI
- [x] Document steps required to create a custom tap and target

### What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)


### Links / references

_Please note that this was taken from GitLab, to be changed accordingly_

Story points: 1.0
Silence pip unless there is an error

As per the 2018-01-14 demo, this was a feedback:

> Pip version 10.0.1 red warning.
> Micael: Should we silence pip?
> Micael: We can either upgrade pip (I don’t like that) or silence the ouput unless there is an error (I like that)

I think we should simple silence pip unless there is a problem.

Make the error YELLOW and not RED when it's a warning.

Story points: 1.0
Consider a private meltano managed `.meltano/dbt/profiles.yml` for dbt

## Background

Currently Meltano creates a not-very robust `profiles.yml` file which attempts to be "one size fits all" but it only supports a few different config options and has several magic environment variables it expects. This is a barrier to new users onboarding and frequently we get "not declared" errors for those environment variables being missing from runtime context.

Pros of status quo:

1. Gives an out-of-box dbt experience for three different generic database configs: postgres, redshift, and snowflake
    - https://gitlab.com/meltano/files-dbt/-/blob/master/bundle/transform/profile/profiles.yml
2. Doesn't require a user to understand the `profiles.yml` format.

Cons of status quo:

1. Most users will end up deleting the extra profiles or modifying the file to make it their own. (Which feels wrong, like you are breaking the connection to Meltano.)
1. There's no support for other databases like Spark or Redshift.
1. Manual changes could get overridden if the files bundle is reinstalled.
1. Following from point 1, it's not clear at all if overriding/modifying the built-in `profiles.yml` is supported - or if there are other consequences of doing so.

## Proposal

1. Similarly to how Meltano pre-creates and manages config.json files and catalog.json files for taps, Meltano could auto-generate a hidden `profiles.yml` file internally to the `.meltano` directory.
1. Meltano can populate values from config in `meltano.yml`, rather than requiring special environment variables to be set by the user.
1. Meltano can pick what kind of dbt adapter to use based on the environment's `primary_loader` config (not yet built).
1. As users get more mature in their usage of dbt, they may _optionally_ provide a path to their own generated profiles.yml file. This could be an optional config option for the `dbt` transformer and/or of the meltano environment itself.
1. The user can optionally provide a profile name for dbt to use - as a config option for the `dbt` transformer.
    - If not set, the dbt profile target can default to the environment name (`prod`, `userdev`, etc.).
1. Since the hidden file would not be version controlled, it can be adapted without breaking existing code. For instance, we could add handling for a Spark profile and we wouldn't need users to reinstall `files-dbt` bundle.

Story points: 8.0
Document pyenv/poetry use in contributor docs

Our getting started guide for contributors isn't super cohesive in how the setup steps for contributing to meltano, the meltano api, or the meltano ui are described, and what the expected dev environment looks like.

1. For example in the "setting up your environment" section we have folks clone/poetry install https://meltano.com/docs/contributor-guide.html#setting-up-your-environment
2. The api dev section makes no further mention of poetry and has you do a vanilla `meltano init` 
3. Similar the UI dev sections also makes no further mention of poetry.

Would it be useful to maybe to include a preamble on working with pyenv/poetry - and then providing general contributor, api, ui dev docs that take that into account? A lot of folks will already have experience with things like pyenv/virtualenv/poetry but it might still be nice to touch on them and clarify their potential use for managing your dev environments.

Story points: 1.0
meltano run ci/cd pipeline project

Something similar to the `meltano/demo-project` running as part of our CI/CD suite but using `meltano run` would be handy. I think initially, it would probably be wise to make it an optional stage (i.e. `allow_failure: true`) - so that failures on that stage don't block our entire pipeline. Another option might be to initially have this be a scheduled pipeline - https://docs.gitlab.com/ee/ci/pipelines/schedules.html

Story points: 4.0
Document Nested Select Examples

Based on https://gitlab.com/meltano/sdk/-/issues/285#note_750983833 where incorrectly configured select criteria for a complex nested case looked like an SDK bug to the user. We should update [the documentation](https://meltano.com/docs/command-line-interface.html#select) to show the example that @edgarrmondragon provided. I believe the solution was that you need to include the highest level attribute before you can select nested attributes (correct me if I'm wrong).

Story points: 1.0
meltano run improvements for closer elt parity

For `meltano run` to be a full replacement of `meltano elt` there's a few small additions that we'll probably need to consider. This originally pop'd up in https://gitlab.com/meltano/meltano/-/issues/2301#note_750941505 during the initial `meltano run` build.

1. ~~Support for incremental jobs~~ (now logged as #3130):
    1. Job ID support via Job ID prefix sourced from a CLI flag or based on the activated meltano env. (https://gitlab.com/meltano/meltano/-/issues/2301#note_750941505)
    1. Top-level `--full-refresh` to ignore saved state and do a full refresh. (https://gitlab.com/meltano/meltano/-/merge_requests/2432#note_757893658)
    1. Top-level `--no-state-update` to not save state. (https://gitlab.com/meltano/meltano/-/merge_requests/2432#note_757893658)
1. Top-level `--force` to ignore locks or indications of already-running jobs. (https://gitlab.com/meltano/meltano/-/merge_requests/2432#note_757893658)
1. A "dry run" option ala `meltano run --dry-run ...` (https://gitlab.com/meltano/meltano/-/issues/2301#note_750941505)

Note: Behind the scenes - cleaning up the ELBContext -> ELTContext relationship and what singer plugins expect probably needs to be part of the Job support work.

Story points: 12.0
Document MacOS specific configuration step for contributing to the UI/API

On MacOS BigSur (and up?) port 5000 is [used by Airplay/Control Center](https://developer.apple.com/forums/thread/682332). For new users setting up a development environment that can cause a somewhat hard to troubleshoot issue because `meltano ui` won't actually be able to start. It's easy enough to work around by [setting alternate bind ports](https://meltano.com/docs/settings.html#ui-bind-port), so we should just explicitly document this in the contributor guide - and flag it for MacOS users.

Long term we have https://gitlab.com/meltano/meltano/-/issues/3088 to try and auto-detect/warn if the port is unaivable, and to change the default port for new projects.

Story points: 1.0
Clarify venv usage by Meltano when inheriting with an environment

Had a user ask today whether or not Meltano re-installed a plugin in a separate venv when a [Meltano Environment](https://meltano.com/docs/environments.html#inheritance) is used. I wasn't sure of the answer and the docs weren't clear.

@edgarrmondragon can you clarify this and make an MR to the docs?

Story points: 1.0
Auto-detect UI port availability and possibly switch default for new projects

On MacOS BigSur (and up?) port 5000 is [used by Airplay/Control Center](https://developer.apple.com/forums/thread/682332). For new users setting up a development environment that can cause a somewhat hard to troubleshoot issue because `meltano ui` won't actually be able to start. It's easy enough to work around by [setting alternate bind ports](https://meltano.com/docs/settings.html#ui-bind-port), so we could just explicitly document this in the contributor guide - and flagging it for MacOS users. 

Would it be worth considering changing the default api port from 5000 (and 8080 for the dev ui) to something like 8080 (and 8181 for the dev ui)?

Story points: 4.0
Update pinned version of target-bigquery

Latest version of adswerve/target-bigquery is [0.11.3](https://github.com/adswerve/target-bigquery/releases), but the plugin in `discovery.yml` is pinned to an earlier version: `git+https://github.com/adswerve/target-bigquery.git@v0.10.2`.

There have been some important bug fixes since that, like https://github.com/adswerve/target-bigquery/issues/9.

cc @pnadolny13 @tayloramurphy

Story points: 1.0
Document how STATE is handled in greater detail

The specifics around handling of 'incomplete state' (state from previous failed jobs) is not included in the [relevant section of the docs.](https://meltano.com/docs/integration.html#incremental-replication-state).

Thread from [Slack](https://meltano.slack.com/archives/CMN8HELB0/p1637186511109700):

> I should know this already, but...  If a loader fails in an ELT job, what happens to the data and the state?  I reviewed the Meltano ELT reference architecture and it does not say, and I have not looked at the source.  Is the main Meltano process actually sitting in the middle and watching the messages from extractor stdout and sending them to loader stdin, and watching stdout of the target for that state to know that the target has processed, and stored, records up to that state?  I've had that assumption all along, but didn't really know.

> I know the feeling. It's one of those things that you really want to know is solid and understand, but I've also often had the feeling I didn't really understand what would happen in exceptional situations.
Taking the code is truth approach - these are the relevant places I think:
https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/target.py#L21
https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/runner/singer.py#L90
As each STATE message flows through, meltano (or optionally delegated to the target it seems) writes the state.
Would be great to have one of the @meltano-engineering guys jump in and correct me, or even some links to detailed design docs! (edited)

Story points: 2.0
Allow users to name a default Environment in `meltano.yml`

## Summary
[//]: # (Concisely summarize the feature you are proposing.)

Let users set the name of the default environment in `meltano.yml`.

## Proposed benefits
[//]: # (Concisely summarize the benefits this feature would bring to yourself and other users.)

Users don't need to pass `--environment` to every command or set the `MELTANO_ENVIRONMENT` variable.

## Proposal details
[//]: # (In as much detail as you are able, describe the feature you'd like to build or would like to see built.)

```yaml
version: 1
default_environment: dev

environments:
- name: dev
  ...
- name: prod
  ...
```

means commands will use `dev` if `--environment` is not passed.

## Best reasons not to build
[//]: # (Will this negatively affect any existing functionality? Do you anticipate any breaking changes versus what may already be working today? Make the counter-argument to your proposal here.)

Can confuse users about which environment will be activated if they use `--environment` and `default_environment` at the same time, but that can easily addressed in the docs and clearly giving precedence to the former.

Story points: 4.0
Support a --log-format short cut

Per @rabidaudio in https://gitlab.com/meltano/meltano/-/merge_requests/2395#note_731245620

> Would it be possible to configure the log format from CLI args? It seems like the way to configure it as designed now is through the log handler files. It seems clunky to ask users to turn on structured logs everywhere just so they show up nicely in Dagster. I suppose our Dagster pipelines could generate an alternate `logging.yaml` file and pass with `--log-config`, would this be the recommended way?

We could definitely add a basic `--log-format=[json|keyvalue|uncolored-console]` flag that switches meltano to use one of our other built-in formatters for the console logger *if* there's no logging config present. 

It would be a nice shortcut for scenarios where folks don't want to supply a more elaborate logging config, and really just want to be able to switch from a human-friendly format to a machine-friendly format.

Story points: 2.0