giseldo/neodataset huggingface dataset:
Object Storage should have a strict TTFB timeout

We had/have an outage related to GCS used by GitLab Pages:

https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=avg(rate(gitlab_pages_httprange_trace_duration_sum%7Brequest_stage%3D%22httptrace.ClientTrace.GotFirstResponseByte%22%7D%5B5m%5D)%2Frate(gitlab_pages_httprange_trace_duration_count%7Brequest_stage%3D%22httptrace.ClientTrace.GotFirstResponseByte%22%7D%5B5m%5D))&g0.tab=0&g1.range_input=2h&g1.expr=sum(gitlab_pages_httprange_open_requests)&g1.tab=0&g2.range_input=2h&g2.stacked=1&g2.expr=sum(increase(gitlab_pages_vfs_operations_total%7Boperation%3D%22Open%22%7D%5B5m%5D))%20by%20(vfs_name%2C%20operation)&g2.tab=0

### TTFB

![image](/uploads/fd84ed1775b3ae7c32735a9bf8c76cfb/image.png)

We clearly see that TTFB sky rockets once we disabled NFS. In regular operation we expect `25ms`, during outage of GCS we were seeing around `20s`, but once we disabled it appears that some requests had TTFB close to "a few minutes".

### Connections

![image](/uploads/24df853cd766d9b48b65962c0464ab49/image.png)

This created a crazy backlog of 6k open connections to GCS. Our timeout on the connection is 30 minutes.

### Summary

This outage showed some aspects to improve in Pages related how Pages handles timeouts, and what timeouts it should use to ensure sane service recovery and reducing amplification pressure due to outage happening elsewhere.

### Proposal

Define TTFB to be no longer than 15s. This should ensure that "sane amount" of requests is opened, and system can fast-reject instead of hang.

```golang
var httpClient = &http.Client{
	// The longest time the request can be executed
	Timeout: 30 * time.Minute,
	Transport: httptransport.NewTransportWithMetrics(
		"httprange_client",
		metrics.HTTPRangeTraceDuration,
		metrics.HTTPRangeRequestDuration,
		metrics.HTTPRangeRequestsTotal,
	),
}
```

We should configure the:

- `TLSHandshakeTimeout time.Duration`, aka before being able to write `request`
- `ResponseHeaderTimeout time.Duration`, aka close to `TTFB`
- `ExpectContinueTimeout time.Duration`, aka similar to `TTFB`

Or, just implement that differently as part of RoundTrip of `httprange` to define a timeout to receiving response (but not yet fully reading it).

More here: https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

Story points: 1.0
The unavailability of GitLab API makes us fallback to disk-serving

Currently, if the GitLab API is unavailable we fallback to using `disk serving`.

This is OK, now, but in a fully cloud-native architecture, this should rather
return a proper `500` type of error.

We should disable `local instance()` and instead return a proper error to client.

```golang
func (d *Domain) resolve(r *http.Request) *serving.Request {
	request, _ := d.Resolver.Resolve(r)

	// TODO improve code around default serving, when `disk` serving gets removed
	// https://gitlab.com/gitlab-org/gitlab-pages/issues/353
	if request == nil {
		return &serving.Request{Serving: local.Instance()}
	}

	return request
}

// ServeFileHTTP returns true if something was served, false if not.
func (d *Domain) ServeFileHTTP(w http.ResponseWriter, r *http.Request) bool {
	if !d.HasLookupPath(r) {
		// TODO: this seems to be wrong: as we should rather return false, and
		// fallback to `ServeNotFoundHTTP` to handle this case
		httperrors.Serve404(w)
		return true
	}

	request := d.resolve(r)

	return request.ServeFileHTTP(w, r)
}
```

Story points: 1.0
Fix TestVFSFindOrCreateArchiveCacheEvict flaky test

I found this flaky test locally, I believe this line https://gitlab.com/gitlab-org/gitlab-pages/-/blob/6cba2f2875856a7b4e2d008ca348d7b91098ed35/internal/vfs/zip/vfs_test.go#L120 is causing trouble

```
 go test ./internal/vfs/zip -run TestVFSFindOrCreateArchiveCacheEvict -count 10 -failfast
--- FAIL: TestVFSFindOrCreateArchiveCacheEvict (0.00s)
    vfs_test.go:126:
        	Error Trace:	vfs_test.go:126
        	Error:      	Should not be: &zip.zipArchive{fs:(*zip.zipVFS)(0xc0000b1e90), path:"http://127.0.0.1:60311/public.zip", once:sync.Once{done:0x1, m:sync.Mutex{state:0, sema:0x0}}, done:(chan struct {})(0xc0000b26c0), openTimeout:30000000000, cacheNamespace:"1:", resource:(*httprange.Resource)(0xc00007fe00), reader:(*httprange.RangedReader)(0xc00007ba50), archive:(*zip.Reader)(0xc00007fe80), err:error(nil), files:map[string]*zip.File{"public/":(*zip.File)(0xc000240160), "public/404.html":(*zip.File)(0xc000240370), "public/bad_symlink.html":(*zip.File)(0xc000240790), "public/index.html":(*zip.File)(0xc0002402c0), "public/subdir/":(*zip.File)(0xc000240420), "public/subdir/2bp3Qzs9CCW7cGnxhghdavZ2bJDTzvu2mrj6O8Yqjm3YMRozRZULxBBKzJXCK16GlsvO1GlbCyONf2LTCndJU9cIr5T3PLDN7XnfG00lEmf9DWHPXiAbbi0v8ioSjnoTqdyjELVKuhsGRGxeV9RptLMyGnbpJx1w2uECiUQSHrRVQNuq2xoHLlk30UAmis1EhGXP5kKprzHxuavsKMdT4XRP0d79tie4tjqtfRsP4y60hmNS1vSujrxzhDa":(*zip.File)(0xc0002406e0), "public/subdir/hello.html":(*zip.File)(0xc0002404d0), "public/subdir/linked.html":(*zip.File)(0xc000240580), "public/symlink.html":(*zip.File)(0xc000240630)}}
        	Test:       	TestVFSFindOrCreateArchiveCacheEvict
        	Messages:   	a different archive is returned
FAIL
FAIL	gitlab.com/gitlab-org/gitlab-pages/internal/vfs/zip	0.132s
FAIL
```

Story points: 1.0
Pages 404s with pages_artifacts_archive enabled

We had a Silver Customer ([Zendesk](https://gitlab.zendesk.com/agent/tickets/176158) - internal only) report that they started getting 404s on their Pages project, but it worked when copying in a new project.

That happened a few hours after the `pages_artifacts_archive` rollout percentage was increased.

@ayufan turned off the feature flag and we were able to see the Pages again.

Relevant [Slack thread](https://gitlab.slack.com/archives/C1BSEQ138/p1602582002162400?thread_ts=1602547809.159500&cid=C1BSEQ138) (internal only)

Story points: 3.0
Improve zip cache refreshing for the same domain

We open a bunch of archives each minute: https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=increase(gitlab_pages_zip_opened%5B5m%5D)&g0.tab=0

From slack: https://gitlab.slack.com/archives/C1BSEQ138/p1602144418115300

```
kamil Oct 8th at 7:06 PM
@jaime I think we are done with LRU for time being. Maybe next get archive refresh to be working?
4 replies

kamil  5 days ago
Looking at this graph it appears that we open a bunch of archives each minute: https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=increase(gitlab_pages_zip_opened%5B5m%5D)&g0.tab=0

jaime  1 day ago
do you mean we’re refreshing too often?

kamil  22 hours ago
Yes, we each time we refresh the domain via API a new URL is being returned due to changed auth params. This means that we reload each accessed archive every minute. (edited) 

kamil  22 hours ago
Due to caching, holding 2-3 archives for a given domain at a single time.
```

Story points: 1.0
Add vfs type to access log

From [slack comment](https://gitlab.slack.com/archives/C1BSEQ138/p1602147615122100?thread_ts=1602147179.119100&cid=C1BSEQ138):
> I also think that it would be helpful for us to extend kibana logs with information on how the file was served (if it was ZIP), so we could ELK grep logs and find unique domains being served.

Story points: 1.0
Improve tooling for local Golang development

Local development is not friendly for Go newcomers, decreasing the likelihood of ~"Community contribution"s

- Since go modules was introduced, `make setup` fails trying to remove `.GOPATH/`. $GOPATH is not needed anymore so we should remove this

```sh
$ make setup

rm: .GOPATH:
...
rm: .GOPATH/pkg/mod/sourcegraph.com/sqs/pbtypes@v0.0.0-20180604144634-d3ebe8f20ae4/void.pb.go: Permission denied
```

- `make lint` fails with `ERRO Running error: context loading failed: no go files to analyze`
- Add `make format` for people that run into linter issues , e.g. https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/764364111

Story points: 3.0
Negative cache errored zip archives

Having two intervals:

* positive cache: long-time cache, likely prolonged
* negative cache: short-time cache, likely not prolonged

---
The following discussion from !351 should be addressed:

- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/351#note_418261431): 

    > I believe that we have a TODO item: Should we cache an archive that cannot be read for a long period? Aka. negative cache.
    > 
    > This works very well for positive cache, but also makes us to cache negative cache (errored archive forever).

Story points: 1.0
Ignore `@ hashed` directory when performing disk scan.

In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42461 we store zipped pages site version on the same top directory as other pages content. But add `@hashed` to avoid name collision with namespaces.

IIRC pages currently recusively scans all subdirectories in `shared/pages`, so we need exclude `shared/pages/@hashed`.

It may sound irrelevant for .com, since we disabled this disk scan completely. But it's not true for self-managed instances, and once we [enable feature flag responsible for creating pages deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/245308) this directory will appear on self-managed. This may create performance problem and maybe some security risks(unlikely though).

But maybe pages will just not find `config.json` file there and ignore this directory automatically, but that's need to be tested. :shrug:

Story points: 1.0
Handle extra headers when serving from compressed zip archive

Handle some missing headers lost functionality by using `io.Copy` instead of `http.ServeContent`.

Go src for [`http.ServeContent`](https://github.com/golang/go/blob/ef20f76b8bc4e082d5f81fd818890d707751475b/src/net/http/fs.go#L130) as reference.

## Proposal

1. Handle request headers:
- If-Match
- If-Unmodified-Since
- If-None-Match
- If-Modified-Since
- ~~If-Range~~ can't use this header with compressed files

2. Set in response header:
- Last-Modified

~"backend-weight::2" 
---
The following discussion from !348 should be addressed:

- [ ] @vshushlin started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/348#note_415122700):  (+2 comments)

    > **Question:** from `ServeContent` docs:
    > 
    > > ...sets the MIME type, and handles If-Match, If-Unmodified-Since, If-None-Match, If-Modified-Since...
    > 
    > Do we plan to reimplement MIME types and modified-since logic? Do we need follow up issues for them?

Story points: 3.0
Implement zip cache callbacks

The following discussion from !351 should be addressed:

- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/351#note_414831416): 

    > As a follow-up maybe we would have "a kind of callback" on `zipArchive`?
    > 
    > ```golang
    > func (z *zipArchive) onCacheAdded() {}
    > func (z *zipArchive) onCacheRefreshed() {}
    > func (z *zipArchive) onCacheEvicted() {}
    > ```
    > 
    > We would simply call these methods on these points, and they could update metrics on a will, assuming that if you ever call `onCacheAdded()` the `onCacheEvicted()` will be called as well.

And implement tests https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/351#note_418262162

Story points: 1.0
Make zip cache configurable

A simple in-memory cache was introduced in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/351 and https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/364 with some hard-coded values. These should be configurable.

## Proposal

* Add configuration changes flag to Pages
  - Add a zipCacheConfig type and flags:
  - `zip-cache-expiration`
  - `zip-cache-cleanup`
  - `zip-cache-refresh`
  - `zip-open-timeout`
  - ~~`zip-cache-dataoffset-size`~~
  - ~~`zip-cache-dataoffset-prune-count`~~
  - ~~`zip-cache-readlink-size`~~
  - ~~`zip-cache-readlink-prune-count`~~
* Add documentation
* Add flag to omnibus

---
The following discussion from !351 should be addressed:

- [ ] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/351#note_414788991):  (+2 comments)

    > **question**: are these values good enough? Should we make them configurable in future iterations?

Story points: 3.0
Cache dataOffset and symlink when requesting zip files

Currently `Readlink` and `dataOffset` are evaluated each time, we should cache them to lower the latency when serving from zip.

To ensure that `Readlink` is processed in a predictable time, using minimal amount of requests, especially that in some cases we need to perform symlink traversal.

---

The following discussion from !348 should be addressed:

- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/348#note_412690978): 

    > I wish that we would cache `dataOffset`, since the `file.DataOffset()` each time makes us to fire additional request.

Story points: 1.0
Make httprange http client configurable

In https://gitlab.com/gitlab-org/gitlab-pages/-/issues/448 we introduced an `httprange` package that can download artifacts and support reading ranges from the response body. The `httpClient` used has a hardcoded timeout which should be configurable.

We should either pass an httpClient when initializing a resource or some config flags

## Proposal

- Add configuration changes flag to Pages
- Add documentation
- Add flag to omnibus
- Add flag to charts

---
The following discussion from !333 should be addressed:

- [ ] @vshushlin started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/333#note_408522039): 

    > Do we already have an issue for that? Can we add a link?

Story points: 3.0
Use versioned templates for security scanners

Some security templates keep changing and they break our pipelines when they are merged to the default branch. We should use a versioned template pointing to a stable branch, e.g.

```yaml
inclue:
  - https://gitlab.com/gitlab-org/gitlab/-/raw/13-3-stable-ee/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml
```

---
The following discussion from !343 should be addressed:

- [ ] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/343#note_408774219): 

    > We should use versioned templates

Story points: 1.0
TestClientStatusClientTimeout flaky

https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/718191293

```
=== RUN   TestClientStatusClientTimeout
    client_test.go:303: 
        	Error Trace:	client_test.go:303
        	Error:      	"failed to connect to internal Pages API: Get "http://127.0.0.1:42673/api/v4/internal/pages/status": context deadline exceeded" does not contain "Client.Timeout"
        	Test:       	TestClientStatusClientTimeout
--- FAIL: TestClientStatusClientTimeout (0.06s)
```

Story points: 1.0
Create an `httprange` package to allow loading content from object storage

Source https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/326/diffs#c697a55849ad2b6da28d7b6de7e8effd7860e40b

- `http_range` should create resources that can be read in chunks from a pre-signed URL
- Add metrics and traceln

Story points: 2.0
Abstract `VFS` `Root`

Currently, pages assumes that projects are located in subdirectories `pages_dir/group/subgroup/project/public/index.html`. While in the zip object storage VFS will just have `/public/index.html` in the archive structure.

Based on the work in https://gitlab.com/gitlab-org/gitlab-pages/-/issues/439 we want to abstract some kind of `Root directory` for the project. It will allow us to implement zip serving much easier.

Story points: 1.0
VFS implementation for zip archives

Add `zip` implementation of the VFS interface introduced in https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/324 and !325.

Depends on https://gitlab.com/gitlab-org/gitlab-pages/-/issues/446 and https://gitlab.com/gitlab-org/gitlab-pages/-/issues/448.

- Add a `zip` package that opens archives using `http_range` - source https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/326/diffs#f0eb7ce2a638003d1f5dd027e9b5ce3aa44355e7_0_1
- Implement the `VFS` interface - source https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/326/diffs#21487386d5527c7631ac241c03fb0400516c3839_0_1
- Update metrics acceptance tests for `TestPrometheusMetricsCanBeScraped`

Story points: 2.0
Fix intermittent cache test failures

Job [#658818771](https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/658818771) failed for 43f724fe2886e07cd97799ff0a14a3b98a239ec3:

```plaintext
    --- FAIL: TestResolve/when_retrieval_failed_because_of_resolution_context_being_canceled (0.00s)
        cache_test.go:264: 
            	Error Trace:	cache_test.go:264
            	            				cache_test.go:113
            	            				cache_test.go:253
            	            				cache_test.go:93
            	            				cache_test.go:252
            	Error:      	An error is expected but got nil.
            	Test:       	TestResolve/when_retrieval_failed_because_of_resolution_context_being_canceled
```

Another job https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/658818891

```plaintext
=== RUN   TestResolve/when_item_is_in_long_cache_only
    cache_test.go:207: 
        	Error Trace:	cache_test.go:207
        	            				cache_test.go:113
        	            				cache_test.go:200
        	            				cache_test.go:93
        	            				cache_test.go:199
        	Error:      	Not equal: 
        	            	expected: 0x1
        	            	actual  : 0x0
        	Test:       	TestResolve/when_item_is_in_long_cache_only
```

Retrying a job fixes it https://gitlab.com/gitlab-org/gitlab-pages/-/jobs/658821575

Story points: 1.0
Use exponential back-off for polling status endpoint

The following discussion from !304 should be addressed:

- [ ] @jaime started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/304#note_382039256):  (+2 comments)

    > **question**: should we consider using an exponential back-off approach?
    > 
    > Some sample library https://pkg.go.dev/github.com/cenkalti/backoff/v4?tab=doc#pkg-examples

Story points: 1.0
Preprocess zip archives on load and cache file structure

Preprocess zip archive and cache relevant files:

* cache only relevant list of files with references where to fetch them in an efficient way
* cache into a `map` to have `O(logn)` search

Reference code https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/326/diffs#f0eb7ce2a638003d1f5dd027e9b5ce3aa44355e7_0_34

Depends on https://gitlab.com/gitlab-org/gitlab-pages/-/issues/443

### Considerations

A noticeable change came up during the [profiler demo](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/479#note_423617206) with ~"team::Scalability"  where we saw that `readArchive` is allocating a considerable amount of memory (about 28MB for the top 1%) for `docs.gitlab.com` alone in production. 

[Profiler on GCP](https://console.cloud.google.com/profiler;timespan=1h/gitlab-pages;type=HEAP/inuse_space?project=gitlab-production) (internal) 

![Screen_Shot_2020-10-08_at_3.21.17_pm](/uploads/6a0a161802a5c7b5d2c31b5dcbe4be06/Screen_Shot_2020-10-08_at_3.21.17_pm.png)

[Allocated memory spike](https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1w&g0.stacked=1&g0.expr=max(go_memstats_alloc_bytes%7Binstance%3D~%22web-pages-.*%22%7D)&g0.tab=0) after enabling Zip for 5% of Pages projects on 2020-10-08 ~12:30 UTC

![Screen_Shot_2020-10-12_at_10.52.25_am](/uploads/36f50746826ed854b4d2a2019d6cbad9/Screen_Shot_2020-10-12_at_10.52.25_am.png)

---
<details>
<summary>The following discussion from !299 should be addressed:
</summary>
- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/299#note_377210608): 

    > Oh, yes, yes, yes. It would really help to convert a flat list into:
    > 
    > - `map[string]zip.File`, ideally with `zip.File` having an `offset` as well
    > 
    > And, drop all the ones that are not within `public/`

Cache flat list idea https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/299#note_377210617
</details>

Story points: 1.0
Tech Evaluation of supporting NFS in cloud native deployments as an alternative to object storage for Pages

### Problem to solve 

Support for object storage in Pages is currently looking to arrive early FY22.

This is impacting a few deployments to our `gitlab` helm chart:

1. GitLab.com's [migration to Kubernetes](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/98) (multiple services depend upon Pages - including web and api services)
1. A very large opportunity who requires Pages

One way to potentially solve this, is to introduce shared storage into our Kubernetes deployments.

### Evaluation 

* [ ] Discuss feasibility of this idea with ~"group::distribution", ~infrastructure, and ~Quality 
* [ ] Ask large opportunity if they would be open to / capable of utilizing shared storage in their Kubernetes cluster
* [ ] Implement Pages in the Helm chart and cloud native images (does not look like this is present today) - https://gitlab.com/gitlab-org/charts/gitlab/-/issues/37
  * [ ] Add support for a shared `ReadWriteMany` PV across needed services (Pages, Sidekiq, API, and Web)
* [ ] Run a 50k architecture load test on a cloud native deployment with Pages (will require utilizing something like Rook, Portworx, etc. to provide ReadWriteMany support)
* [ ] Strongly consider utilizing this architecture on GitLab.com, so we can move forward with k8s migration/dogfooding, as well as confirm this Pages architecture works at scale before recommending to our customers
* [ ] Launch official support for Pages in Helm charts using shared storage

Story points: 1.0
Add metrics for zip serving

Zip serving package implements the `type Serving interface`. We should add some metrics to serve from zip archives similar to https://gitlab.com/gitlab-org/gitlab-pages/-/issues/365 so that we can track and compare serving times from archives vs disk.

- ZipServingOpenArchivesTotal
- ZipServingFilesPerArchiveCount

https://gitlab.slack.com/archives/C1BSEQ138/p1601113581019900

> I think we should for OS access have a histogram of timings of different request stages, like: connection, tls handshake, response write, response to the first byte, response read. This would allow us to understand how responsive is OS on different stages. (edited) 

>Also, for caches we should have hit/miss metric, and amount of cache entries :slightly_smiling_face:

Story points: 1.0
Tech Evaluation: Extract the Reader logic from disk serving into its own package so it can be reused

Serving from disk uses the [`type Reader struct`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/master/internal/serving/disk/reader.go#L19) to try to find and resolve paths from the file system structure. 

This logic can be reused to serve content from zip archives https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/299/diffs#580d90130ae2083bae1a5eda81a8947e0b57d443_0_51

Story points: 1.0