giseldo/neodataset huggingface dataset:
Keep track of dev dependencies in go.mod

When running `make setup`, the dev tools will be installed into the project's local `.GOPATH`. However, this overwrites the `go.mod` and `go.sum` files with a reference to those tools. The `make deps-check` rule will fail **If the changes are committed by mistake**.

Using a `tools.go` file is a common pattern to keep track of dev dependencies inside the `go.mod` file.

The [release-cli tools.go](https://gitlab.com/gitlab-org/release-cli/-/blob/master/cmd/release-cli/tools.go) has an example of how to do it. We also reference https://marcofranssen.nl/manage-go-tools-via-go-modules/:

> To ensure my tool dependencies are not removed and can leverage the Go Modules, I create a file tools.go. In this file I will list all my tool dependencies using an import statement.

To do:

- Add a file called `tools.go` to the root directory and add the dev/tool dependencies.
- `make deps-check` will add an entry to `go.mod`, consider version locking to a desired release.
- Commit the changes.

Story points: 1.0
SAST scanning fails intermittently

Example of a job where the SAST scanning fails

Job [#562578827](https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/562578827) failed for 98c479549ef6e4017fa052a4d5f19535e70b6d2e:

```
docker: error during connect: Post http://docker:2375/v1.40/containers/create: dial tcp: lookup docker on 169.254.169.254:53: no such host.
```

From [Slack thread](https://gitlab.slack.com/archives/CETG54GQ0/p1590023829443200)

> @fcatteau license-scanning works fine so this might be related to sast, and in that case the legacy SAST orchestrator (Docker-in-Docker)

>by the way, the CI configuration implicitly enables the Docker-in-Docker orchestrator for SAST even though the no-DinD is now the default
https://gitlab.com/gitlab-org/security/gitlab-pages/-/blob/master/.gitlab-ci.yml#L63
see https://gitlab.com/gitlab-org/gitlab/-/issues/218541

>the important difference between sast and license-scanning is that the former explicitly calls docker run whereas the latter sets the image.name in its job definition. this might explain why License Scanning works. I'm able to pull registry.gitlab.com/gitlab-org/security-products/sast:2 though

>  I suggest you override the rules of the *-sast jobs triggered for this project, and don't override sast anymore. could you try that in a MR? also, I suggest you share this with #g_secure-static-analysis

> actually you only need to override secrets-sast and gosec-sast . see https://gitlab.com/gitlab-org/security/gitlab-pages/-/jobs/548248714#L192

Story points: 1.0
Add Dangerbot to Pages

Pages does not currently have a Dangerbot and therefore does not suggest reviewers for MRs. In theory any engineer working on Golang should be able to do reviews, so there would be a large pool of reviewers.

The maintainer pool in Pages is quite small presently, but this is [being worked on](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/7701), with the intention of developing more maintainers within ~"group::release".

Additional reviewers would be added to `team.yml`

```yml
  projects:
    gitlab-pages: reviewer
```

Story points: 2.0
Tech Evaluation for option B of Serve custom 404.html file for namespace domains

https://gitlab.com/gitlab-org/gitlab-pages/-/issues/183

Open the research issue for implementing the approach `B` and prioritize it. It's really tricky to implement and basically only `custom 404s for namespace level projects, e.g. username.gitlab.io` are broken, they are not broken on custom domains or on `username.gitlab.io/myprojectname`.

Tasks

* [ ]  Update proposal for linked issue 

Story points: 1.0
Use custom .golangci.yml file with improved lint rules

The following discussion from !264 should be addressed:

- [ ] @krasio started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/264#note_333342106):  (+2 comments)

    > Is this to avoid name clash with the `client` package?

--- 

We use [automatic linting](https://docs.gitlab.com/ee/development/go_guide/#automatic-linting) via `golangci-lint` with a default set of relaxed rules. To improve the code style we should add a custom `.golangci.yml` file enabling a set of useful linters.

We can use [GitLab's Runner `.golangci.yml`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/.golangci.yml) for consistency or use as a base for the set of rules we want to apply to Pages.


cc @krasio @vshushlin @sean_carroll

--- 

## Linters

From [GitLab's Runner `.golangci.yml`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/.golangci.yml)

| Linter | Enable in iteration # | 
| ------ | :--------------------:|
| bodyclose | 2 |
| deadcode | 2 |
| dogsled | 2 |
| goconst | 1 |
| gocyclo | 1 |
| goimports | 1 |
| golint | 1 |
| gosec | 1 |
| gosimple | 1 |
| govet | 1 |
| ineffassign | 2 |
| misspell | 2 |
| structcheck | 2 |
| typecheck | 2 |
| unconvert | 2 |
| unused | 2 |
| varcheck | 2 |
| whitespace | 2 |

### Other linters to consider

List of supported [linters](https://golangci-lint.run/usage/linters/) by `golangci-lint`

| Linter | Description |
| ------ | :----------:|
| wsl | improve empty line spacing (annoying but makes the code more readable) |
| interfacer | Linter that suggests narrower interface types |
| lll | long lines |
| unparam |  Reports unused function parameters |
| gocritic | The most opinionated Go source code linter |
| gochecknoinits | This will help with #342 |

---

## MRs

- [x] Add .golangci.yml linter configuration !286 
- [x] Enable more linters https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/287

Story points: 2.0
Remove `gitlab-server` fallback to `auth-server` and `artifacts server`

Currently `auth-server` and `artifacts-server` may be used instead of `gitlab-server` command line flag. That was done for a seamless upgrade.

Deprecation was announced, so we can just remove it in %"14.0"

From @igorwwwwwwwwwwwwwwwwwwww's comment https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/270#note_334218639 :

> Alternatively, I think it might be good to remove this fallback logic.
> 
> From an operational perspective, I found it very surprising that `gitlab-server` inherits from both auth server as well as artifacts server.
> 
> I would find the inverse *much* more intuitive. But even having to specify all the params would be acceptable.
> 
> So if we can verify that the main consumers of this (probably just omnibus?) are always specifying `gitlab-server` explicitly, it may be worth simply requiring it to be specified without fallback.

We need to:

- [x] Mark auth-server as deprecate in Omnibus(might be already done)
- [x] Make a release deprecation post. draft:
```
We're changing the behaviour of some configuration options of GitLab Pages. If you run GitLab Pages via Omnibus based package, no action required. Otherwise, make sure to set [`gitlab-server` parameter and remove `auth-server`](https://docs.gitlab.com/ee/administration/pages/).
```


Later we may consider removing `artifacts-server` and use `gitlab-server`(or better: new parameter introduced in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/276) and appending path.

Story points: 2.0
Meltano Invoke/Select does not invalidate catalog cache after reinstalling a tap

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label and verify the issue you're about to submit isn't a duplicate.

If you are submitting an issue with a tap, please include:

- account details
- target details
- entities selected  with meltano select (if you have selected any entities), as the bug may be related to a specific entity
- the full elt command you are running
- full output of the meltano elt command. Logs can get pretty long, so you can add the full log as a snippet in the Meltano project and add a link in the issue.
--->

### What is the current *bug* behavior?
Taps are using cached properties for `meltano invoke/select` even if the tap's properties have changed after `meltano install`.



### What is the expected *correct* behavior?
When running `meltano install`, a tap's properties cache should be invalidated so that `meltano invoke/select` should effectively "rediscover" the tap's properties.



### Steps to reproduce
```
# install all taps
1. meltano install
2. meltano invoke tap-custom-tap
# update tap-custom-tap repo with some new schema properties
3. meltano install tap-custom-tap
4. meltano invoke tap-custom-tap
```


### Relevant logs and/or screenshots
After running `meltano install extractor tap-s3-toast`
```
> meltano --log-level=debug invoke tap-s3-toast
[2021-03-04 21:28:53,618] [1|MainThread|root] [DEBUG] Creating engine <meltano.core.project.Project object at 0x7f510102d160>@sqlite:////projects/.meltano/meltano.db
[2021-03-04 21:28:54,642] [1|MainThread|root] [DEBUG] Created configuration at /projects/.meltano/run/tap-s3-toast/tap.config.json
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find tap.properties.json in /projects/.meltano/extractors/tap-s3-toast/tap.properties.json, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find tap.properties.cache_key in /projects/.meltano/extractors/tap-s3-toast/tap.properties.cache_key, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|root] [DEBUG] Could not find state.json in /projects/.meltano/extractors/tap-s3-toast/state.json, skipping.
[2021-03-04 21:28:54,643] [1|MainThread|meltano.core.plugin.singer.tap] [DEBUG] Using cached catalog file
```

### Possible fixes
According to @DouweM and my [Slack conversation](https://meltano.slack.com/archives/C013EKWA2Q1/p1614892532050300), `meltano invoke/select` should have some logic to invalidate caches. Other options are removing the cache when running `meltano install` either by default or with an optional cli flag.



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 4.0
Add Superset as a plugin

- https://superset.apache.org/
- https://github.com/apache/superset
- https://pypi.org/project/apache-superset

```bash
meltano add analyzer superset

meltano invoke superset db upgrade # Run automatically?
meltano invoke superset fab create-admin
meltano invoke superset init # Run automatically?
meltano invoke superset run
```

Meltano can manage Superset configuration (https://superset.apache.org/docs/installation/configuring-superset) by allowing values to be set for the keys in https://github.com/apache/superset/blob/master/superset/config.py, automatically generating `superset_config.py`, and pointing Superset there by using the `SUPERSET_CONFIG_PATH` env var. Users should also be able to set `SUPERSET_CONFIG_PATH` (or `meltano config superset set config_path <path>`) themselves to use their own config file.

Ideally, Meltano would also be able to inject database connection strings corresponding to loaders directly into Superset so that these don't need to be managed in two places: https://superset.apache.org/docs/databases/installing-database-drivers, https://superset.apache.org/docs/databases/postgres. Possibly through the `DB_CONNECTION_MUTATOR` setting? https://github.com/apache/superset/issues/9045

Story points: 8.0
Let plugin `pip_url` take into account Python version

The `pip_url` for `airflow` is currently `apache-airflow==1.10.14 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1.10.14/constraints-3.6.txt`, using a constraints file built specifically for Python 3.6, regardless of the version that's actually used.

With Airflow 1.10.14, this is OK, because the constraints file is actually valid on 3.7 and 3.8 as well, but this is not the case for Airflow 2.0.1's https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.6.txt.

This can be seen in https://gitlab.com/meltano/meltano/-/merge_requests/2032 and pipelines https://gitlab.com/michelrado/meltano/-/jobs/1026252711 (3.7) and https://gitlab.com/michelrado/meltano/-/jobs/1025856502 (3.8).

One way to solve this would be to let a `PYTHON_VERSION` environment variable to referenced from `pip_url`, like so:

```yaml
pip_url: 'apache-airflow==2.0.1 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-${PYTHON_VERSION}.txt'
```

Story points: 4.0
Document how to manage incremental replication state without a persistent system database

As I wrote on Slack:

>>>
Note that there's already a workaround for state management with ephemeral system databases: since `meltano elt` (and `meltano schedule run`) can take a `--state` argument to run with a specific state, and can be run with `--dump=state` to dump the current state instead of running the underlying pipeline, a serverless ELT wrapper can:
1. download the state from S3
2. run the pipeline with `--state=<downloaded state>`
3. dump the current state (from the local ephemeral system database) to a file with `--dump=state > new-state.json`
4. upload the new state to S3
>>>

Story points: 4.0
Pipelines UI should let an arbitrary cron expression be chosen as a schedule interval

`meltano schedule` already supports arbitrary cron expressions, e.g. `0 */6 * * *` to mean "every 6 hours": https://meltano.com/docs/command-line-interface.html#schedule

The UI should support this as well, both when creating a new pipeline and editing an existing one.

At the same time, we should show these intervals correctly: https://gitlab.com/meltano/meltano/-/issues/2527

Update (2021-11-29): To provide a good user experience, I (AJ) think we should also preview the "next 10 occurrences" (or similar) so the user can ensure the cron expression is valid and meets their intended schedule.

Story points: 8.0
Remove flakehell now that it's no longer being maintained

https://github.com/life4/flakehell was added in https://gitlab.com/meltano/meltano/-/merge_requests/1970, but the maintainer has now archived the project: https://github.com/wemake-services/wemake-python-styleguide/issues/1817.

Since it's no longer being maintained, we should move away from it and start using `flake8 --diff` directly.
The docs at https://wemake-python-stylegui.de/en/latest/pages/usage/integrations/legacy.html will likely come in handy as well.

Story points: 4.0
Environment Variable Substitution doesn't work with array values

### What is the current *bug* behavior?
Environment variables can't be used in the `meltano.yml` file for arrays.



### What is the expected *correct* behavior?
The yaml parser should traverse arrays to substitute environment variables. It would also be nice to add custom yaml variable support to the parser. (e.g. `custom_var: var` -> `database: '{{ custom_var }}'`)



### Steps to reproduce
- In a `meltano.yml` file try to use an environement variable in an array.


### Relevant logs and/or screenshots
```
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    config:
      host: host
      user: root
      database: ${DB_NAME}
    select:
    - ${DB_NAME}-users.id
    - ${DB_NAME}-users.email
    metadata:
      ${DB_NAME}-users:
        replication-method: FULL_TABLE
```
`database: ${DB_NAME}` works because it's an object. But it doesn't work in the select because it's an array.

### Possible fixes
Not sure, sorry.



### Further regression test
_Ensure we automatically catch similar issues in the future_

- [ ] Write additional adequate test cases and submit test results
- [ ] Test results should be reviewed by a person from the team

Story points: 8.0
Let project plugin definitions and config be defined in multiple individual YAML files instead of `meltano.yml`

As I wrote in https://gitlab.com/meltano/meltano/-/issues/2206#note_388496082:

>>>
I think we could also allow users to define their project's taps and targets in their own YAML files, instead of having everything live in `meltano.yml`, since the `select`, `metadata`, `schema`, and `config` properties can grow quite large.

These files could live at `<type>/<name>.yml` in the project, e.g. `extractors/tap-example.yml`:

```yaml
variant: meltano
pip_url: git+https://gitlab.com/meltano/tap-example.git
select: ...
metadata: ...
schema: ...
config: ...
```

We wouldn't need to support variant subdirectories in this case, since only a single variant of a plugin should exist in a project anyway, which also goes for custom plugins.
>>>

------------------

## [Update] Latest spec description:

In `meltano.yml` specifying an include directory:

```yaml
project_id: ...
include_paths:
  - connectors/*.yml
  - schedules/*.yml
...
plugins: ...
schedules: ...
```

Or explicitly with exact file paths:

```yaml
project_id: ...
include_paths:
  - connectors/data-team.yml
  - connectors/business-team.yml
  - schedules/prod-refreshes.yml
...
plugins: ...
schedules: ...
```

And the companion files can be defined as:

```yaml
# connectors/data-team.yml
plugins:
  extractors:
    - name:...

  loaders:
    - name:...
```

```yml
# connectors/business-team.yml
plugins:
  extractors:
    - name:...

  loaders:
    - name:...
```

```yml
# schedules/prod-refreshes.yml
schedules:
  ...
```

Additional info:
- The included files have the same spec as `meltano.yml`, with the exception that some top-level entries like `project_id` might not be permitted.
- Rather than establish a priority between file declarations, collisions would be treated as errors, for instance if two files both declare an `extractor` with the name `tap-gitlab`.

Story points: 8.0
Consider renaming extractor "entities" and "attributes" to "streams" and "properties" for consistency with Singer

This would prevent confusion and inconsistent/interchangeable use of the Meltano-specific and Singer-specific term in docs or conversation.

Meltano users don't technically need to be familiar with underlying Singer concepts, and "entities" and "attributes" are arguably more user-friendly and less implementation-specific, but especially at this stage this is unrealistic since users will still often need to debug (and fix) taps and read their repositories and documentation, which use the Singer terms.

If Meltano were to at one point add support for a different open source EL standard that uses different terms, using "streams" and "properties" may no longer make sense, but that's a bridge we can cross then.

Story points: 4.0
`docs` job fails with vague `ERROR: Job failed: exit code 1`

See https://gitlab.com/meltano/meltano/-/jobs/841670357:

```bash
<snip>
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/mstile-144x144.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/meltano-diagram.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/apple-touch-icon.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/mstile-310x150.png
Cleaning up file based variables
ERROR: Job failed: exit code 1
```

The failing command is [`$REMOTE_EXEC "mkdir -p $SSH_BACKUP_DIRECTORY; tar zcvf $SSH_BACKUP_DIRECTORY-$(date +%Y-%m-%dT%H:%M).tar.gz $SSH_DIRECTORY/$SSH_WWW_DIRECTORY"`](https://gitlab.com/meltano/meltano/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml#L25), where `$REMOTE_EXEC` is `ssh -o StrictHostKeyChecking=no $SSH_USER_DOMAIN -p$SSH_PORT`.

An earlier run (https://gitlab.com/meltano/meltano/-/jobs/841365856) logged those exact same output lines, but then was able to continue on to the next command:

```bash
<snip>
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/mstile-144x144.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/meltano-diagram.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/apple-touch-icon.png
/home/u894-pruvsekvb4qh/www/meltano.com/public_html/mstile-310x150.png
$ $REMOTE_EXEC "cd $SSH_DIRECTORY && find ./$SSH_WWW_DIRECTORY -mindepth 1 -maxdepth 1 -not -name blog -not -name '.' -exec rm -rf  '{}' \;"
$ scp -o stricthostkeychecking=no -P$SSH_PORT -r public/* $SSH_USER_DOMAIN:$SSH_DIRECTORY/$SSH_WWW_DIRECTORY
Saving cache for successful job
Creating cache default...
docs/node_modules/: found 25033 matching files and directories 
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/7603319/default 
Created cache
Cleaning up file based variables
Job succeeded
```

Story points: 2.0
Nested properties in discovered catalog are interpreted as `inclusion: automatic` even if their parent property is `inclusion: available`

See https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/catalog.py#L284.

As a result, these nested properties will always show up as `automatic` in `meltano select --list --all`, cannot be deselected using `meltano select --except`, and end up having metadata `inclusion: automatic, selected: true` in the generated catalog file that's passed to the tap.

In reality, if a parent property is discovered as `inclusion: available`, changing `selected` or `inclusion` on an individual subproperty usually won't make a difference since taps typically only check the `selected` metadata for top-level properties, not nested ones.

To resolve this, then, we can have Meltano assume `inclusion: automatic` on top-level properties that lack metadata, but not on nested properties, (assuming their parent property does have metadata). In `meltano select --list --all`, nested properties without discovered(!) `inclusion` metadata shouldn't be listed at all, since they can't actually be individually selected. If a nested property does have discovered `inclusion` metadata, we can list it. 

We could automatically have a top-level `selected` rule cascade down to its properties, but let's not get into that for now.

---

As a temporary workaround, the following `metadata` rule can be added to a plugin definition to flip these nested `inclusion: automatic` metadatas to `inclusion: available`:

```yaml
extractors:
- name: tap-facebook
  # ...
  metadata:
    '*':
      '*.*':
        inclusion: available
```

Story points: 8.0
Allow extractor Singer messages to be transformed using Python before passing them to loader (Stream Maps)

We could support [`transformer` plugins](https://meltano.com/docs/plugins.html#transformers) other than `dbt`, that would follow extraction rather than loading, and would transform an extractor's output stream of Singer messages (`SCHEMA`, `RECORD`, etc), before they're streamed into the loader.

The two types of transformers could be distinguished using an extra, e.g. `type: {etl,elt}` or `{follows,acts_on,transforms}: {extractor,loader}`

We could support both pip packages, and local executable files, using either the `pip_url` or `executable` plugin setting.

Story points: 12.0
Preserve comments and flow style when programmatically modifying `meltano.yml` (Ruamel)

PyYAML doesn't support this currently (https://github.com/yaml/pyyaml/issues/90), but https://pypi.org/project/ruamel.yaml does.

Note:

Our usage of the Canonical class may also be a factor we need to consider in our implementation. Noted by @kgpayne below: https://gitlab.com/meltano/meltano/-/issues/2177#note_708071594

Story points: 20.0
Create Helm chart to deploy Meltano to Kubernetes

The Helm chart should ideally run the Meltano schedule via Kubernetes orchestration without Airflow if possible.

Minimum Goal:
The minimum goal will to be to provide a way to run a single `meltano elt` pipeline on a schedule with a Kubernetes job.

Ideal Outcome:
The ideal outcome would be to support both running a project specific docker image and the default Meltano docker image in both readonly, as well as read-write mode to support the broadest possible user base. Each scheduled pipeline created using `meltano schedule` and available through `meltano schedule list --format=json` would result in its own scheduled `meltano elt` job.

Story points: 12.0