giseldo/neodataset huggingface dataset:
Support for single page applications route all requests to index.html.

## Problem to Solve 

When routing requests to Gitlab Pages, if there is no default index.html, the request renders an error that does not permit SEO or support of SPA on GitLab Pages. 

## Solution 

Route all requests to GitPages to /index.html by:
*  implementing a `route single page app to index.html` to a checkbox in **Pages > Settings**
* Enable SPAs to serve any accompanying files so that it works
* Ensure existing files are served before trying to serve `index.html`

## Update 2021-07-20
Closing this issue in favor of https://gitlab.com/gitlab-org/gitlab-pages/-/issues/57. We aim to enable SPAs via a new `.gitlab-pages.yml`

Story points: 4.0
Follow-up from "Add auth-cookie-session-timeout flag" - In Auth constructor use options struct

The following discussion from !834 should be addressed:

- [ ] @proglottis started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/834#note_1156512951): 

    > nit, not directly related to your change, but this constructor has enough arguments now that it's probably worth extracting an options struct.

Story points: 2.0
`cache_pages_domain_api` feature flag cause failures in staging

### Summary

During the rollout of `cache_pages_domain_api` feature flag, the gitlab pages qa tests on staging started to fail.

After some investigation with `@vshushlin`, we found out that the application settings are different between rails nodes (where the cache is created) and sidekiq nodes (where the cache is invalidated). Since we us a hash for the application settings in the cache key, the sidekiq job never finds the cache key to invalide it.

Related:
- https://gitlab.com/gitlab-org/gitlab/-/issues/376332+
- https://gitlab.com/gitlab-org/gitlab/-/issues/364127+

### Possible fixes

```json:table
{
    "items" : [
{
"option": "🅰️",
"description": "Instead of using the application settings hash in the cache key we could insert the hash in the cached value and validate it when reading the cache. This way the app settings is not part of the cache key, so it should get invalidated in the sidekiq nodes.",
"cons": "validating the cache manually based on the cached value."
},
{
"option": "🅱️",
"description": "Instead of using the application settings hash in the cache key we could have a two layers cache. The main cache key would save the app settings hash, which would be the key to the payload itself.",
"cons": "two reads on redis to find the cached value."
}
]
}
```

Story points: 3.0
Support HSTS on GitLab Pages on per domain basis (also on gitlab.com)

## Problem to Solve

Some users want to enforce no access to their web content without HTTPS/certificates. 

This can be done with [HTTP Strict Transport Security (HSTS) policy](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security), but we do not currently support enabling this for GitLab Pages sites on GitLab.com.

## Solution 

Allow users to enable HSTS for Pages on GitLab.com.

Story points: 5.0
Session timeout after 10 minutes

### Summary

We use gitlab-pages for several internal website hostings and our users experience annoying session timeouts due to the short lifetime of the gitlab-pages cookie of 10 minutes. As an example, a single page web app can't load further assets (e.g. images) after the session timeout and a (hard) refresh in the browser is necessary.

The `authSessionMaxAge` constant is hardcoded in [`internal/auth/auth.go`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/14e310074668254398d13f74e647dd16df2cb12c/internal/auth/auth.go#L40):

```go
authSessionMaxAge      = 60 * 10 // 10 minutes
```

It seems that this was introduced with https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/178.

So we have a few questions:
- Why was the timeout set to exactly 10 minutes? Is there a reason for this (e.g. a standard)?
- If there is no specific reason, would it be possible to increase this value?
- Or would it be possible to make this value configurable?


WDYT?

If desired, we would be happy to support you as ~"Leading Organization"  with a ~"Community contribution".

### Example Project

<!-- If possible, please create an example project here on GitLab.com that exhibits the problematic 
behavior, and link to it here in the bug report. If you are using an older version of GitLab, this 
will also determine whether the bug is fixed in a more recent version. -->

### What is the current *bug* behavior?

There's a session timeout after 10 minutes.

### What is the expected *correct* behavior?

The duration to a session timeout is higher or can be configured.

### Goal

Make the timeout configurable:

1. [x] Add a new flag to Pages e.g. `auth-cookie-session-timeout` with default value of `10m` :arrow_right: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/834
1. [x] Add the flag to Omnibus, see [sample MR](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6144/diffs) :arrow_right: https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6495
1. [x] Add the flag to the Helm charts, see [sample MR](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2338/diffs) :arrow_right: https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2838
1. [x] Update the admin documentation, see [sample MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/89780/diffs) :arrow_right: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/102996

Story points: 3.0
GCS ranged requests sometimes fails with 400

<!---

Please read this!

**NOTE**: Please check https://gitlab.com/gitlab-org/gitlab/-/issues/331699 if you had issues upgrading to 14.0+

Before opening a new issue, make sure to search for keywords in the issues (including closed ones)
- https://gitlab.com/gitlab-org/gitlab-pages/issues

and verify the issue you're about to submit isn't a duplicate.
--->


### Summary

In GitLab.com, we recently ran into an [incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7727) where a small number of Pages hosts failed with 500 errors. Digging into it we found that this resulted from internal error where ranged requests to GCS were failing with "400 Bad Request".

This behavior seems recent (started Sepetmeber 8th), judging by graphs from GCS:
![image](/uploads/795e5836674f380e1ada61233dd81ff8/image.png).

Kibana logs doesn't provide much info beside the error itself, so perhaps we may need to consider logging the response body of 400 responses.

<!-- DO NOT CHANGE -->

Story points: 3.0
Flaky TestDomainResolverError race

I've seen an increase in race test failures for `TestDomainResolverError` recently, I think it's worth investigating a bit more.

Job [#2647463990](https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/2647463990) failed for be06885fdd167a523dd8e1e103bcbb23bb32db61:

Story points: 1.0
Move Gitlab Pages documentation menu in the sidebar one level up

### Summary

Gitlab Pages contribution documentation was recently moved to the docs site. I think it deserves to be a bit more discoverable/easier to access. For that, I propose to move it to its own *contributing* submenu.
| before | after |
|--------|-------|
| <img alt="SCR-20220616-l99" src="https://gitlab.com/gitlab-org/gitlab-docs/uploads/76a348a3f1c32c69d18dfb674aa7780d/SCR-20220616-l99.png" width="300"/> | <img alt="image" src="https://gitlab.com/gitlab-org/gitlab-docs/uploads/0abb0e10552412569fa43e60ffc03de1/image.png" width="300"/> |

Story points: 1.0
Remove doc/development in favor of https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/pages/index.md

<!---

Please read this!

**NOTE**: Please check https://gitlab.com/gitlab-org/gitlab/-/issues/331699 if you had issues upgrading to 14.0+

Before opening a new issue, make sure to search for keywords in the issues (including closed ones)
- https://gitlab.com/gitlab-org/gitlab-pages/issues

and verify the issue you're about to submit isn't a duplicate.
--->


### Summary

### Steps to reproduce

<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->

### Example Project

<!-- If possible, please create an example project here on GitLab.com that exhibits the problematic 
behavior, and link to it here in the bug report. If you are using an older version of GitLab, this 
will also determine whether the bug is fixed in a more recent version. -->

### What is the current *bug* behavior?

<!-- Describe what actually happens. -->

### What is the expected *correct* behavior?

<!-- Describe what you should see instead. -->

### Relevant logs and/or screenshots

<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
 as it's tough to read otherwise. -->

### Output of checks

<!-- If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com -->

### Possible fixes

<!-- If you can, link to the line of code that might be responsible for the problem. -->

<!-- DO NOT CHANGE -->

Story points: 1.0
Investigate and configure bundle on CI to disallow any changes to Gemfile.lock

This is a ~"corrective action" of sirt-2266.

Determine whether this project (https://gitlab.com/gitlab-org/gitlab-pages) uses `bundle` or `bundle install` in CI. Per the Rubygems.org advisory, we should use either the `frozen` or `deployment` options as defense-in-depth to mitigate supply chain attacks. This project needs to use the `frozen` or `deployment` options if not yet.

## More background

See https://github.com/rubygems/rubygems.org/security/advisories/GHSA-hccv-rwq6-vh79:

> Using Bundler in --frozen or --deployment mode in CI and during deploys, as the Bundler team has always recommended, will guarantee that your application does not silently switch to versions created using this exploit.

Note that the `deployment` option installs gems to `vendor`, which we may not want. So `frozen` will usually be the smaller change.

Note that:

> \[DEPRECATED\] The `--frozen` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local frozen 'true'`, and stop using this flag

Story points: 1.0
Add early return and tests for internal/handlers/https

The following discussion from !735 should be addressed:

- [ ] @vshushlin started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/735#note_922396637): 

    > I would add 
    > 
    > ```golang
    > if !redirect {
    > 	return handler
    > }
    > 
    > and tests for this handler. But let's move it to the follow-up issue. This is already a nice improvement :thumbs-up:

Story points: 1.0
Infer artifacts-server from internalGitlabServerFromFlags() unless it's set explicitly

### Summary

We have a lot of config options pointing to the gitlab server. We use them for different reasons:

1. getting domain information
2. redirecting to authentication
3. fetching OAuth token
4. proxying artifacts

In the simplest case, it should be possible to infer all of them from a single URL.

Note:
1. some of these URL work in the `internal` network (e.g. OAuth token, gitlab domain information)
2. while others should be public(e.g. OAuth redirects), because we can redirect the user to this URL

So we should have at least 2 parameters:
- `gitlab-server`
- `gitlab-internal-server`

And all of the rest can be inferred from this.

Right now `artifacts-server` should always be set in the config, I suggest we infer if from `internalGitlabServerFromFlags()` unless it's set.

**Optional:** mark `artifacts-server` as deprecated an remove it on the next major release.

## Implementation details

```golang
func artifactsServerFromFlags() string {
	if *artifactsServer != "" {
		return *artifactsServer
	}

	return internalGitlabServerFromFlags() + "/api/v4"
}
```

Story points: 1.0
Revamp gitlab-pages development docs

<!---

Please read this!

**NOTE**: Please check https://gitlab.com/gitlab-org/gitlab/-/issues/331699 if you had issues upgrading to 14.0+

Before opening a new issue, make sure to search for keywords in the issues (including closed ones)
- https://gitlab.com/gitlab-org/gitlab-pages/issues

and verify the issue you're about to submit isn't a duplicate.
--->


### Summary

While setting up my gitlab-pages environment I got confused with some information on `doc/development.md`. It was also confusing to have some duplicated information in https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/pages.md.

I propose to have all the information regarding gitlab-pages development in one place (a Single Source Of Truth - SSOT). For that, I:

1. Revamped the `gitlab-pages/doc/development.md` to have all the required information to start a gitlab-pages developing environment (this MR)
1. If this MR is approve, I'll open another MR in the GDK repository to point the `GDK/doc/howto/pages.md` to the `gitlab-pages/doc/development.md`.

Story points: 2.0
Arbitrary protocol redirection

<!---

Please read this!

**NOTE**: Please check https://gitlab.com/gitlab-org/gitlab/-/issues/331699 if you had issues upgrading to 14.0+

Before opening a new issue, make sure to search for keywords in the issues (including closed ones)
- https://gitlab.com/gitlab-org/gitlab-pages/issues

and verify the issue you're about to submit isn't a duplicate.
--->


### Summary

<!-- Summarize the bug encountered concisely. -->

GitLab pages can be used to redirect to arbitrary protocols in the authentication flow.


### Steps to reproduce

<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->

Consider the following URL:

```
https://projects.gitlab.io/auth?domain=mailto://gitlab-com.gitlab.io?body=OMGWTF&state=aaa
```

It will, after the login redirect to `gitlab.com` redirect the user to their mail client.

![image](/uploads/44a20d4331d7cc8ee9699884c47d8bc7/image.png)

This might be potentially used e.g. on mobile devices to exfiltrate authentication token via custom URL handler.

### Example Project

<!-- If possible, please create an example project here on GitLab.com that exhibits the problematic 
behavior, and link to it here in the bug report. If you are using an older version of GitLab, this 
will also determine whether the bug is fixed in a more recent version. -->

### What is the current *bug* behavior?

<!-- Describe what actually happens. -->

Redirect to arbitrary protocols is possible.

### What is the expected *correct* behavior?

Redirect should only be possible to `https` or `http` URLs.
<!-- Describe what you should see instead. -->

### Relevant logs and/or screenshots

<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
 as it's tough to read otherwise. -->

### Output of checks

<!-- If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com -->

### Possible fixes

<!-- If you can, link to the line of code that might be responsible for the problem. -->

<!-- DO NOT CHANGE -->

Story points: 2.0
Unable to download large files from gitlab.io

### Summary

If you try to download a large file from gitlab.io the connection is closed after about 37s. This is probably a regression of the recent security fix. 

### Steps to reproduce

Try to download a large file over a slow connection or try to rate limit your connection (`wget --limit-rate` works to reproduce this issue). This affects f.e. https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo. (see https://gitlab.com/paulcarroty/vscodium-deb-rpm-repo/-/issues/77) If you try to download the mentioned `codium_1.66.0-1648720116_amd64.deb` (80Mb) and the download requires more than about 37s the connection is closed and the download is not finished.


## TODO

- [x] change the default to 5 minutes https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/722
- [x] make backports to versions we backported security fix to
   - 1.51: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/723
   - 1.54: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/724
   - 1.56: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/725
- [x] make these parameters configurable in charts and omnibus
   - charts: https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2504
   - omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6029
- [x] log them in http://gitlab.com/gitlab-org/gitlab-pages/blob/4b1afecbb6ae1886bfd3a31f256909ca2770bce4/internal/config/config.go#L331-L331
- [x] remove the hot fix from https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1697
   - https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1731

Story points: 3.0
Remove rate limits feature flags

### Summary

We rolled out rate limits for gitlab pages, so now we can remove feature flags guarding it.

SO we need to:
1. remove 4 relevant feature flags from https://gitlab.com/gitlab-org/gitlab-pages/-/blob/a8be317a96f2dfb4ab30e338daee2b51833c2322/internal/feature/feature.go#L12-L12 
1. remove `enforce` flag from https://gitlab.com/gitlab-org/gitlab-pages/-/blob/master/internal/ratelimiter/ratelimiter.go#L48, and all tests using it
1. clean our production to not mention these feature flags: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/96a3bf149c1cd2847f29c3ed666c480e6b8d15b7/releases/gitlab/values/values.yaml.gotmpl#L449 , https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/96a3bf149c1cd2847f29c3ed666c480e6b8d15b7/releases/gitlab/values/gprd.yaml.gotmpl#L97 , and other files, **grep the repository for each of them**
1. remove mentions of feature flags in https://docs.gitlab.com/ee/administration/pages/#rate-limits

Story points: 2.0
TLS security for GitLab Pages metrics endpoints

Refer https://gitlab.com/groups/gitlab-org/-/epics/7479

GitLab components report metrics via Prometheus, and sometimes start a pprof listener to aid live profiling. Either of these may be open and listening when running GitLab in production, as they are on GitLab.com

1. Inventory which endpoints report to Prometheus
1. Determine of those endpoints are already or can be TLS-secured
1. Secure each unsecured endpoint

~"devops::release" ~"group::release" ~"Category:Pages"

Story points: 5.0
Intermittent 404s with gitlab-org.gitlab.io due to "domain does not exist"

### Summary

@marcel.amirault mentioned that he was seeing intermittent 404s while visiting https://gitlab-org.gitlab.io/gitlab-roulette/ today.

https://log.gprd.gitlab.net/goto/e5236f50-82bc-11ec-a649-b7cbb8e4f62e shows that the `domain does not exist` error pops up quite frequently for this host:

![image](/uploads/81fd4b665c5aded40fe716eb759cbe7f/image.png)

The issue seems distributed across multiple pods:

![image](/uploads/082536dceb7b114b7a8e247abbc4b629/image.png)

Curious why these URLs seem to be involved?

![image](/uploads/520b15b310d4e7bc88608728cf42568b/image.png)

https://gitlab-org.gitlab.io/trello-power-up/scripts/api.js should exist.

<!-- Summarize the bug encountered concisely. -->

### Steps to reproduce

<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->

### Example Project

<!-- If possible, please create an example project here on GitLab.com that exhibits the problematic 
behavior, and link to it here in the bug report. If you are using an older version of GitLab, this 
will also determine whether the bug is fixed in a more recent version. -->

### What is the current *bug* behavior?

<!-- Describe what actually happens. -->

### What is the expected *correct* behavior?

<!-- Describe what you should see instead. -->

### Relevant logs and/or screenshots

<!-- Paste any relevant logs - please use code blocks (```) to format console output, logs, and code
 as it's tough to read otherwise. -->

### Output of checks

<!-- If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com -->

### Possible fixes

<!-- If you can, link to the line of code that might be responsible for the problem. -->

<!-- DO NOT CHANGE -->

~"devops::release" ~"group::release" ~"Category:Pages"

Story points: 5.0
feat: make server shutdown timeout configurable

The following discussion from !664 should be addressed:

- [x] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/664#note_815813893): 

    > // TODO: make this timeout configurable

# Proposal

Add a new config flag `server-shutdown-timeout` of type `time.Duration`.

- [x] Add to Pages server
- [x] Add to Omnibus config (example https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5832/diffs)
- [x] Add to GitLab Charts (example https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2338)
- [x] Document in the admin docs (example https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77969)

Story points: 4.0
refactor: revert back to archive/zip once go1.17 is no longer supported

### Summary

Due to a bug in go1.17, we had to introduced a forked version of the `archive/zip` package: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/646.

This bug is fixed in go1.18, so eventually, we can revert back to using `archive/zip` in place of `gitlab.com/gitlab-org/golang-archive-zip` once support for go1.17 has been dropped.

<!-- DO NOT CHANGE -->

~"devops::release" ~"group::release" ~"Category:Pages"

Story points: 1.0
Wildcard redirects break Let's Encrypt integration

From https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72069: 

> The Let's encrypt automatic integration for gitlab pages is not working when using wildcard redirects.
> 
> I.e. `/* /index.html 200` in _redirects file for pages.

This happens because https://gitlab.com/gitlab-org/gitlab-pages/blob/5e86747b6287381e2f23afe837ede1820876cf8d/internal/acme/acme.go#L33-L33 checks if project has a file in it, and we handle redirects, so our acme middleware doesn't redirect to gitlab.

I see a few ways of fixing this:

1. stop redirecting any `"/.well-known/acme-challenge/*"` files
1. remove this `if domain.ServeFileHTTP(w, r)` check completely. This was originally done to allow users to manually implement their integration with Let's Encrypt. People were doing this before we had our one integration.
1. In the API response for domain add `redirect_for_acme_challenges: true` and replace `if domain.ServeFileHTTP(w, r) {` with `if !domain.RedirectForAcmeChallenges {`
1. In the API response return the actually acme challenges like `{acme_challenges: [{path: "/.well-known/acme-challenge/somehting", value: "somevalue"}]}`. Currently, we rely on the main gitlab instance being public and reachable from Let's Encrypt, so we just redirect to it. If we go with the last option, we can get rid of this requirement. 

However, last 2 options will also slow down the Let's Encrypt integration, as we update domain config cache not that often.

Actually, I'm(@vshushlin) in favor of the first option - it shouldn't break anything for existing users, as not many people use redirects yet. And it doesn't slow down the Let's Encrypt integraiton.

Story points: 1.0
Replace GetCompressedPageFromListener, GetPageFromListenerWithCookie and GetProxiedPageFromListener with GetPageFromListenerWithHeaders

In https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/594/diffs#b0462893f1f2c58c31c518d3cfb09feb2cb3dc57_402_402 we introduced GetPageFromListenerWithHeaders.

Now we can replace GetCompressedPageFromListener, GetPageFromListenerWithCookie and GetProxiedPageFromListener with it. They all are very similar and differ only be headers.

~"devops::release" ~"group::release" ~"Category:Pages"

Story points: 1.0