giseldo/neodataset huggingface dataset:
Make internal/lru metrics optional

### Summary

Currently, https://gitlab.com/gitlab-org/gitlab-pages/blob/247bd7ba2fd9139711218c6a42ed03c551f958d9/internal/lru/lru.go#L30-L30 makes metric parameters required.

So in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/594#note_703313432 we needed to make them required for rate-limiter too.

If these parameters were optional:
* it may be easier to test without need for things like https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/594/diffs#16df871fa23e530eb52c7b56cc4ea5ce8a5e5997_108_113 (unless we want to test metrics specifically)
* `ratelimiter.New` interface would be cleaner

So I suggest:
* Allow `nil` for metrics in https://gitlab.com/gitlab-org/gitlab-pages/blob/247bd7ba2fd9139711218c6a42ed03c551f958d9/internal/lru/lru.go#L30-L30. Or use functional options for them.
* Convert metrics in ratelimiter.New to functional options as well

**This is a very low priority and just nice to have refactoring**

<!-- DO NOT CHANGE -->

~"devops::release" ~"group::release" ~"Category:Pages"

Story points: 1.0
Enable rate limit per source IP without dropping requests

Follow-up 3 https://gitlab.com/gitlab-org/gitlab-pages/-/issues/490#note_676655191

Add configuration flags to Pages, integrate with the `ratelimit` package and add acceptance tests. The flags disable the rate limiter by default so this needs to be enabled in our environments manually. That means 1 MR for staging and another one for production.

Follow approach described in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/575#note_675965682 where we deploy the code and start collecting metrics as-if-enabled to understand the impact of this functionality.

- [ ] pre-prod environments, blocked by https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/2083
- [ ] prod, blocked by  https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5706
- [ ] create a dashboard in Grafana or Kibana

Story points: 3.0
GitLab Pages Source IP Rate Limiting

Follow-up 2 https://gitlab.com/gitlab-org/gitlab-pages/-/issues/490#note_676655191

- Add a `ratelimit` package with support for source IP rate limit
- Add middleware
- Add metrics

### Release notes
GitLab Pages can become quite popular. When too many users are trying to access the Page at the same time, this can lead to an outage.

In this release, GitLab is introducing rate limiting per source IP. This will make Pages hosted on gitlab.com more stable. For users with a self hosted GitLab instance, you can also limit the number of concurrent requesters.

Story points: 3.0
Investigate and test sentry integration

### Summary

Seems like Pages is not reporting any errors to Sentry at all https://sentry.gitlab.net/gitlab/gitlab-pages/.

- We ignore the error when initializing the error tracker in https://gitlab.com/gitlab-org/gitlab-pages/-/blob/master/main.go#L28 so we might be blindly assuming it’s working, when in fact is not.
- There is a way to deploy a local instance of Sentry for testing purposes and integrate with the GDK https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues/748
- Consider implementing some sort of catch-all error reporting.
- Verify the DSN set in https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base-fe-web-pages.json#L32 is correct

Story points: 3.0
Replace defer with t.Cleanup in tests

At least split into 2 MRs

- unit tests
- acceptance tests

But smaller chunks would be great and easier to review


The following discussion from !546 should be addressed:

- [ ] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/546#note_654658851):  (+1 comment)

    > **suggestion**: thinking if we should use `t.Cleanup` and extract this into the `testhelpers`
    > 
    > ```diff
    > git diff --color=always --exit-code
    > diff --git a/internal/auth/auth_test.go b/internal/auth/auth_test.go
    > index d03407a52..d2f219c83 100644
    > --- a/internal/auth/auth_test.go
    > +++ b/internal/auth/auth_test.go
    > @@ -15,6 +15,7 @@ import (
    > 
    >  	"gitlab.com/gitlab-org/gitlab-pages/internal/request"
    >  	"gitlab.com/gitlab-org/gitlab-pages/internal/source"
    > +	"gitlab.com/gitlab-org/gitlab-pages/internal/testhelpers"
    >  )
    > 
    >  func createTestAuth(t *testing.T, internalServer string, publicServer string) *Auth {
    > @@ -211,7 +212,7 @@ func testTryAuthenticateWithCodeAndState(t *testing.T, https bool) {
    >  	require.Equal(t, true, auth.TryAuthenticate(result, r, source.NewMockSource()))
    > 
    >  	res := result.Result()
    > -	defer res.Body.Close()
    > +	testhelpers.Close(t, res.Body)
    > 
    >  	require.Equal(t, http.StatusFound, result.Code)
    >  	require.Equal(t, "https://pages.gitlab-example.com/project/", result.Header().Get("Location"))
    > diff --git a/internal/testhelpers/testhelpers.go b/internal/testhelpers/testhelpers.go
    > index 3ec97a79c..46d4932ca 100644
    > --- a/internal/testhelpers/testhelpers.go
    > +++ b/internal/testhelpers/testhelpers.go
    > @@ -2,6 +2,7 @@ package testhelpers
    > 
    >  import (
    >  	"fmt"
    > +	"io"
    >  	"mime"
    >  	"net/http"
    >  	"net/http/httptest"
    > @@ -68,6 +69,15 @@ func ToFileProtocol(t *testing.T, path string) string {
    >  	return fmt.Sprintf("file://%s/%s", wd, path)
    >  }
    > 
    > +// Close a closer like response.Body as part of testing.T.Cleanup
    > +func Close(t *testing.T, c io.Closer) {
    > +	t.Helper()
    > +
    > +	t.Cleanup(func() {
    > +		require.NoError(t, c.Close())
    > +	})
    > +}
    > +
    >  // Getwd must return current working directory
    >  func Getwd(t *testing.T) string {
    >  	t.Helper()
    > ```
    > 
    > WDYT?

Story points: 2.0
Disable jailing mechanism by default

### Release notes
We disable the jailing mechanism for GitLab Pages by default. The main reason for this is that [many users complain that it causes a lot of problems 
after upgrading to 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). It also lost it relevance since we stopped serving directly from the 
disk starting from 14.0. You can still [enable jailing back](LINK TO https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65791/diffs ONCE IT'S 
MERGED) if disabling it will break something for you. In that case please reach to us on the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331699).

### Background

Starting from GitLab 14.1 the [jailing/chroot mechanism is disabled by default](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/589).
If you are using API-based configuration and the new [Zip storage architecture](#zip-storage)
there is nothing you need to do.

If you run into any problems please [open a new issue](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/new)
and enable the jail again by setting the environment variable:

1. Edit `/etc/gitlab/gitlab.rb`.
1. Set the `DAEMON_ENABLE_JAIL` environment variable to `true` for GitLab Pages:

   ```ruby
   gitlab_pages['env']['DAEMON_ENABLE_JAIL'] = "true"
   ```


Disabling the jail will hopefully fix a bunch of issues related to DNS resolution when running Pages inside a Docker container and ease the transition to the new Pages architecture.


Per @vshushlin suggestion https://gitlab.com/gitlab-org/gitlab-pages/-/issues/589#note_618738893

- Add `daemon-enable-jail` flag to Pages available for source installations
- For Omnibus installations, you can enable this by setting the following environment variable in your `/etc/gitlab/gitlab.rb` file

```rb
gitlab_pages['env']['DAEMON_ENABLE_JAIL'] = "true"
```

Story points: 2.0
Internal API access drops URL path component

I am using the GitLab CE Docker images for 13.12 and run GitLab Pages on a [separate server](https://docs.gitlab.com/ee/administration/pages/#running-gitlab-pages-on-a-separate-server).  My setup still uses disk based configuration and all pages are accessible.  I have `gitlab_pages['inplace_chroot'] = true` and set `external_url` to point to my GitLab instance.

Preparing for an upgrade to 14.0 on a test setup I ran into an issue contacting the GitLab server's API.  This seems to be caused by the fact that my GitLab instance is hosted on a URL like `http://server.example.internal/gitlab`.

On 13.12, I see the following in the GitLab Pages logs
``` console
==> /var/log/gitlab/gitlab-pages/current <==
{"level":"info","msg":"Checking GitLab internal API availability","time":"2021-07-01T01:03:21Z"}
{"error":"failed to connect to internal Pages API: Get \"http://server.example.internal/api/v4/internal/pages/status\": dial tcp: lookup maniac.machine.easy on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address","level":"warning","msg":"attempted to connect to the API","time":"2021-07-01T01:03:21Z"}
```
Note the absence of the `/gitlab` path component in the URL!

After I copied `/etc/resolv.conf` to an `etc/` directory directly below `gitlab_rails['pages_path']`, that changed to
``` console
==> /var/log/gitlab/gitlab-pages/current <==
{"level":"info","msg":"Checking GitLab internal API availability","time":"2021-07-01T01:20:45Z"}
{"error":"failed to connect to internal Pages API: HTTP status: 404","level":"warning","msg":"attempted to connect to the API","time":"2021-07-01T01:20:45Z"}
```
For the above GitLab Pages log entry, I see the following in the logs on my GitLab server
``` console
==> /var/log/gitlab/nginx/gitlab_access.log <==
10.11.55.227 - - [01/Jul/2021:01:20:45 +0000] "GET /api/v4/internal/pages/status HTTP/1.1" 404 1576 "" "Go-http-client/1.1" 1.99
```
confirming that the `/gitlab` path component is ignored.

I have tried setting `gitlab_pages['internal_gitlab_server'] = http://server.example.internal/gitlab` but that didn't make a difference.

Despite all of the above, I am able to access pages using 13.12.  So far so good :thinking:  
However, when I add `gitlab_pages['domain_config_source'] = "gitlab"` to my 13.12 configuration, all I get is `502` HTTP status error pages :sob:  

Seeing that 14.0 requires access to the internal API, it seems I will loose GitLab Pages functionality if I upgrade :worried:

Story points: 1.0
Add flag to disable ACME challenge

Currently Pages depends on [`-gitlab-server` to enable support for the ACME challenge middleware](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/39c7efd7a9045cfde6ca230e5586d8ae1842ddb0/app.go#L517). 
This means it's enabled by default now that the API is on by default. We should have a flag to allow users to disable the middleware if needed.

```go
	if config.GitLab.Server != "" {
		a.AcmeMiddleware = &acme.Middleware{GitlabURL: config.GitLab.Server}
	}
```

## Proposal 

- Add `-disable-acme-middleware` flag to `internal/config/flags.go`
- Add `disable_acme_middleware` to Omnibus default to `nil`

Story points: 2.0
Add Permissions-Policy: interest-cohort=() header to Pages sites hosted on GitLab.com

### Background

This issue is the same as https://gitlab.com/gitlab-org/gitlab/-/issues/327904, but for GitLab Pages instead of GitLab itself.

### Proposal

Begin sending the `Permissions-Policy: interest-cohort=()` header by default for Pages sites.

For now, we'll scope this change to Pages sites hosted on GitLab.com, and not self-managed Pages.

### Why?

- [FLoC has some worrying privacy concerns](https://www.eff.org/deeplinks/2021/03/googles-floc-terrible-idea)
- [GitHub Pages recently blocked FLoC](https://github.blog/changelog/2021-04-27-github-pages-permissions-policy-interest-cohort-header-added-to-all-pages-sites/)

### Additional info

GitHub only enabled this new header for non-custom sites:

> Pages sites using a custom domain will not be impacted

I'm not exactly sure why they made this decision. I think we should send this header by default for _all_ Pages sites, unless there's a convincing reason not to.

### Technical proposal

* Add `Permissions-Policy: interest-cohort=()` to the [config CustomHeaders](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/4ff61e374a7dbfa51fee91799f11197e0f8d53f7/internal/config/config.go#L251) on startup
* Add a boolean flag `-disable-floc-header` (or `-disable-permissions-policy-header`) to https://gitlab.com/gitlab-org/gitlab-pages/-/blob/master/internal/config/flags.go -> when `true` we don't attach that header as above
* Add the option to Omnibus
* Update admin docs

Story points: 3.0
Handle multiple errors in config validation

The following discussion from !465 should be addressed:

- [ ] @hswimelar started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/465#note_560875755):  (+1 comment)

    > This might be a good chance to either use non-fatal logging and return an error at the end. This way, multple problems with the config can be reported all at once, so the user may understand all the work they need to do, rather than having to fix the config one problem at a time.
    > 
    > https://github.com/hashicorp/go-multierror works really well here, but I'm not a maintainer, so I don't feel very confident recommending you to add a third-party dependency. Maybe something do in a commit you can easily revert, if you decide to do down this path.

Story points: 1.0
Allow to disable local disk type of access

Since Pages can run in a fully remote type of access we should
allow disabling disk access completely.

Maybe the simplest way to achieve that would be to specify `-enable-disk=false`,
or switch to specifying an array of `-allowed-paths=[]` instead that would superseed `-pages-root`.

This would be needed to ensure that we fully disable a disk-type of accesses for cases where it is not needed,
like CNG.

Proposal:

- Introduce `-enable-disk`, when false Pages will not need to scan the `pages-root` directory. However, a `pages-root` value is still needed for daemonizing into a jail or chroot.

Story points: 2.0
Introduce a switch for `disk` serving that would allow this type of access being disabled

Currently, we expose `disk` serving as a way to access data.

We should introduce a switch that would disallow usage of that feature with a toggle.
This toggle ideally:
- should be off by default with %"14.0"
- should be off by default for every CNG installation

```
- use-legacy-storage=true
```

Story points: 2.0
Pages "auto" configuration source doesn't work as expected

After `13.7` some people started to have problems with gitlab-pages. Most common symptom: pages are available 50% of the time.

The problem is only present if people have multiple servers running both pages and `gitlab-rails`(see below)

### Consider this setup
* client has gitlab instance "gitlab.example.com"
* and multiple servers running both `gitlab-rails` and `gitlab-pages` simultaneously

### How "auto" currently works
* During startup `gitlab-pages` checks if the API is available multiple times until it gets `200 OK` in response. Which it eventually does because there is a load balancer for "gitlab.example.com" and eventually `gitlab-pages` hits the same server as one it's running on.
* Then `gitlab-pages` remembers that `API is functioning` and will serve `502` errors if it isn't.

### Workaround
* run steps 8-10 in https://docs.gitlab.com/13.7/ee/administration/pages/index.html#running-gitlab-pages-on-a-separate-server
* or explicitly set `gitlab_pages['domain_config_source'] = "disk"` as described in https://docs.gitlab.com/ee/administration/pages/#gitlab-api-based-configuration (not recommended, because in %14.0 this option will be removed and you'll need to do the first option anyway)

### Suggested fix
change "auto" behavior: if we ever fail to access API with unauthorized error - fallback to disk. (do not do this if `gitlab` is set explicitly).

Story points: 2.0
Flaky acceptance tests due to port already in use

Since we split the acceptance tests execution in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/403, we now get a bunch of errors due to the ports being already in use. 

Retrying the jobs one by one in a failing pipeline seem to be enough to make the tests pass.

Sample job https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/912690661
```
msg="could not create socket" error="listen tcp [::1]:37000: bind: cannot assign requested address"
```

A potential solution is to use TCP port 0 and let the OS assign an available port. However, there is no easy way to communicate the port that was assigned to the Pages acceptance tests. This causes [`WaitUntilRequestSucceeds`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/21066de52d7f7af759bdf2395694c935110da1bb/test/acceptance/helpers_test.go#L155) to wait until timeout to get a response from the binary currently running.

Sample faling pipeline
https://gitlab.com/gitlab-org/gitlab-pages/-/pipelines/237350515

---

## Update 2020-01-07

IPv6 seems to be disabled in some `gitlab-org-docker` runners https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12258.

Story points: 1.0
Make gitlab client cache configurable

The `internal/source/gitlab/cache/` package is currently semi hardcoded using the [`defaultCacheConfiguration`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/5e5ce293e7eb529f4443198b8cd8bb4d9a1ff418/internal/source/gitlab/cache/cache.go#L11). This makes it hard to write acceptance tests that can execute faster. For example https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/393#note_463829311.

We should introduce extra flags that will allow configuring the cache's behaviour for resolving a domain's configuration.

## Default configuration

```go
var defaultCacheConfig = cacheConfig{
	cacheExpiry:          10 * time.Minute,
	entryRefreshTimeout:  60 * time.Second,
	retrievalTimeout:     30 * time.Second,
	maxRetrievalInterval: time.Second,
	maxRetrievalRetries:  3,
}
``` 

## Implementation

1. Add the following flags to Pages and override `defaultCacheConfig`

  - gitlab-cache-expiry -> The maximum time a domain's configuration is stored in the cache (default: 600s)
  - gitlab-cache-refresh -> The interval at which a domain's configuration is set to be due to refresh (default: 60s)
  - gitlab-cache-cleanup -> The interval at which expired items are removed from the cache (default: 60s)
  - gitlab-retrieval-timeout -> The maximum time to wait for a response from the GitLab API per request (default: 30s)
  - gitlab-retrieval-interval -> The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1s)
  - gitlab-retrieval-retries -> The maximum number of times to retry to resolve a domain's configuration via the API (default: 3)


2. Move `cacheConfig` to `internal/config/`
3. Add flags to Omnibus
4. Documentation updates

The first part of https://gitlab.com/gitlab-org/gitlab-pages/-/issues/507 is done so we can work on this

## What do these settings do

Users can configure these settings when using [API-based configuration](https://docs.gitlab.com/ee/administration/pages/#gitlab-api-based-configuration) to modify the cache behavior for a domain's resolution. The recommended default values are set inside GitLab Pages and should only be modified if needed.

Some examples:

- Increasing `gitlab-cache-expiry` will allow items to exist in the cache longer. This setting might be useful if the communication between Pages and GitLab Rails is not stable or the content served by Pages does not change frequently.
- Increasing `gitlab-cache-refresh` will reduce the frequency at which GitLab Pages requests a domain's configuration from GitLab Rails. This setting might be useful for content that does not change frequently.
- Decreasing `gitlab-retrieval-retries` will report issues between Pages and Rails more quickly. However, they can be transient failures rather than real issues.

Story points: 4.0
Refactor `Error`s to strings that aren't used as `error`

Follow up to https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/399#note_455909360

@jaime started a discussion https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/399#note_455909360:

> If we are using them as error messages rather than errors, then we should probably change them to just string constants or even inline them in the log line itself. There's no point in defining an error that is not going to be returned or used as value, WDYT?
> 
> e.g. define the messages as strings
> 
> ```go
> const (
>	createArtifactRequestErrMsg = "Failed to create the artifact request"
>	artifactRequestErrMsg       = "Failed to request the artifact"
> )
> ```

Story points: 1.0
Use CorrelationID middleware in Pages

Since Pages does not appear to use CorrelationIDs are present (it should!)

The following discussion from !397 should be addressed:

- [ ] @andrewn started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/397#note_452561607):  (+2 comments)

    > @jaime would you mind reviewing please?

---

1. Add the correlation ID to Pages, including a `propagate-correlation-id` flag.
1. Add flag to Omnibus `gitlab_pages['propagate_correlation_id'] to Omnibus
1. Update Pages admin documentation 

---

# Propagating the correlation ID

Setting the `propagate_correlation_id` in Omnibus will allow installations behind a reverse proxy generate and set a correlation ID to requests sent to GitLab Pages. When a reverse proxy sets the header value `x-request-id`, the value will be propagated in the request chain. Users [can find the correlation ID in the logs](https://docs.gitlab.com/ee/administration/troubleshooting/tracing_correlation_id.html#identify-the-correlation-id-for-a-request).

Story points: 3.0
Refactor config flags and consolidate in internal/config/ package

- Move all config flags to `internal/config/`
- Refactor config struct so it can be called from any other package

The following discussion from !392 should be addressed:

- [ ] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392#note_450695359): 

    > Move all config flags to `internal/config/`

Story points: 2.0
Make `/pages` to be an actual `artifacts-path`

We decided that `file://` being used for ZIP will return a full absolute path as seen from Rails perspective.

However, when doing `chroot` in `pages` we mount `pages-root` under `/pages`. We should instead ensure
that in chroot env the absolute path to access resource is exactly the same, to allow the `file://` returned by
Rails to function exactly the same.

This is required for https://gitlab.com/gitlab-org/gitlab-pages/-/issues/485.

Story points: 1.0
Support splat (wildcard) redirects

In https://gitlab.com/gitlab-org/gitlab-pages/-/issues/24, a simple redirect mechanism was implemented https://docs.gitlab.com/ee/user/project/pages/redirects.html.

It would be great if we also supported splat (wildcard) redirect support. According to the [Netlify docs](https://docs.netlify.com/routing/redirects/redirect-options/#splats):

```
/news/*  /blog/:splat
```

This would redirect paths like `/news/2004/01/10/my-story` to `/blog/2004/01/10/my-story`.

It would mean a lot for the docs site, since we want to move away from the `/ee` prefix and use something like `/gitlab`.

```
/ee/*  /gitlab/:splat
```

### Release Notes

Details to be added

Story points: 2.0