giseldo/neodataset huggingface dataset:
Export billing info

**As a** purchase / ordering employee
<br>**I want** to download all financial information about the use of my services 
<br>**so** I can create invoice for my customers

**:desktop_computer: How to demonstrate / e2e test(s):**

1. go to the 'Activity' page
1. click on the download button
1. open file with Numbers on my shiny Mac

**:white\_check\_mark: Acceptance criteria**

- Filename: YYYYMMDD-activiteit-per-organisatie.csv
- see csv drawing below for columns  
   Note: see warning below for actual info
- Group by: service -> organisation -> month
- Documentation about 'Enable transaction logs' should include instructions on how to specify log database URL to NLX Management
- Only include transactions for you inway in report
- When unable to connect to the DB -> export should be an empty CSV

**:link: Reference information**

![csv-export](/uploads/b1ce3ad60fc736fa63a28e04cbbe6b79/Untitled-2021-02-08-1345.png)  
> source: https://excalidraw.com/#json=4645701698977792,v1uHInLjAMIyq8Q48_coXQ

:warning: The above rows are not completely accurate anymore. For each organization/service it should list:

- 1 line for initial setup cost
- 1 line per month if montly fee
- 1 line per month per # requests

![_todo__Organization_activity_Copy_4](/uploads/2491713205c5489942fb75ad0da8be66/_todo__Organization_activity_Copy_4.png)

State where there is no transaction log configured:<br>
![image](/uploads/c8cccf650c0157dd77214b8043b070e2/image.png)

Story points: 5.0
Document NLX in production

**As a** developer
<br>**I want** know how to setup NLX in my production environment
<br>**so** that I can run NLX in production

**:desktop_computer: How to demonstrate / e2e test(s):**

1.

**:white\_check\_mark: Acceptance criteria**

1. Create a NLX in production section on docs.nlx.io
1. Make the destinction between Try NLX and NLX in production as clear as possible
1. Improve 'Request a production certificate' clearly describe and number all steps needed to get your production certificate. 
1. Add a description on how to generate the internal PKI certificates
1. Add drawing of connections that use internal PKI and PKI-O created by @hugoboos 

**:link: Reference links**

1.

Story points: 8.0
Document helm charts

**As a** developer
<br>**I want** know how I can use the NLX helm charts
<br>**so** that I am able to install NLX on my Kubernetes cluster

**:desktop_computer: How to demonstrate / e2e test(s):**

1.

**:white\_check\_mark: Acceptance criteria**

1. Add a readme to each chart describing the possible values. (https://github.com/jetstack/cert-manager/blob/master/deploy/charts/cert-manager/README.template.md)
1. Make the charts available as a helm repository (https://helm.sh/docs/topics/chart_repository/)


**:link: Reference links**

1.

Story points: 5.0
Validate claims in authorization middleware

Currently, in the `OnlyAuthenticated` middleware (`management-api/pkg/oidc/authenticator.go:106`) we only validate if the `user` session is set and if it contains a valid user object. This is not sufficient for secure authorization.  

**Acceptance criteria**  

1. We should validate the claims and at least perform these checks:

- Is the `nbf` (Not Before) in the future?
- Is the `exp` (Expiry Date) in the past?

These checks can be performed stateless.

2. Save refresh token on auth. Use refresh token after current access token has expired.

Story points: 3.0
Add authorisation to Management UI

**As a** 
<br>**I want** _goal_
<br>**so** _purpose_


**:white\_check\_mark: Acceptance criteria**

1. Store roles for users in the NLX Management database
1. Show error message when user exists in OpenID connect but not in NLX Management
1. In the future we would like to map groups in OpenID Connect to roles in NLX Management
1. There should be atleast a admin role. 
1. A user can be added using the command-line. When adding the user you can specify the ID(email) and role. Eg `nlx-management-api create-user --username dillen --role admin`  

**Acceptance criteria**
1. When the user does not have the 'admin' role authentication should fail
1. When something in the authentication flow fails the user should see what went wrong

**:link: Reference links**

1.

Story points: 8.0
Polling: outgoing access request & access proof on directory detail page

**As a** user
<br>**I want** the detail page to update with the latest outgoing access request and access proof data
<br>**so** I know what's going on

**:desktop_computer: How to demonstrate / e2e test(s):**

1.

**:white\_check\_mark: Acceptance criteria**

1. poll every 3000ms
1. updates outgoing access request & access proof data
1. UI will update to latest state

**tech impl**

1. subscribe to polling manager
1. update mobx

Story points: 2.0
Polling: incoming access requests & access grants in service detail view

**As a** user
<br>**I want** to be notified when there are new incoming access requests or access grants when I'm looking at the service detail page
<br>**so** that I can see the new data on clicking the blue pill

**:desktop_computer: How to demonstrate / e2e test(s):**

1. open service detail page
1. do an access request in another tab
1. see the blue pill appear
1. click blue pill to update list

**:white\_check\_mark: Acceptance criteria**

1. poll every 3000ms
1. design in epic
1. clicking the blue pill will update the list

**:link: Reference links**

1.

**:unicorn: Designs**

Specs:<br>
`background-color: $colorBrand4; //#36C6FF`<br>
`color: $colorTextInverse; //$colorPaletteGrey900`<br>
`margin-bottom: -$spacing03; //of iets anders waardoor ie 8px lager staat dan de content`<br>

Icon:<br>
![arrow-down-line.svg](/uploads/fb770900dcfea4fa85f61481a5d984e3/arrow-down-line.svg)

<br><br>
![Group_113](/uploads/c87645c933bb68d32a7e950a5b5a8468/Group_113.png)

![Screen_Recording_2020-12-02_at_16.05.18](/uploads/6757ccab5213d6c75e008908b6966661/Screen_Recording_2020-12-02_at_16.05.18.mov)

Story points: 3.0
Polling: incoming access requests on service overview page

**As a** user
<br>**I want** incoming access requests to automatically appear when I'm looking at the service overview page
<br>**so** I know about the actual state of incoming access requests



**:desktop_computer: How to demonstrate / e2e test(s):**

1. go to service overview page
1. add an access request from other tab
1. see the UI update

**:white\_check\_mark: Acceptance criteria**

1. poll every 3000ms
1. design in epic

**Technical implementation**

1. add new endpoint: `/api/v1/services/stats`  
   Let's discuss the format of the returned JSON
1. use polling manager
1. from react -subscribe to manager

**Details**

@publicJorn and I discussed that the format should be:

```
{
  "test-service-1": {
    "incomingAccessRequestCount": 1
  },
  "test-service-2": {
    "incomingAccessRequestCount": 2
  }
}
```

This is neat because then we can add other statistics to the same endpoint in the future.

Story points: 3.0
Show warning when removing organization inway

**As a** user
<br>**I want** to see a warning when removing organization inway
<br>**so** I am warned that things will go wrong

**:desktop_computer: How to demonstrate / e2e test(s):**

1. go to settings
1. remove organization inway

**:white\_check\_mark: Acceptance criteria**

1. In settings, when resetting organization inway (becomes null) show warning:  
   "Bij het verwijderen van de organisatie inway kunnen huidige toegangsverzoeken niet meer worden afgehandeld en kan je geen nieuwe meer ontvangen"

Story points: 1.0
Docs: try NLX management config mismatch

## Situation

On the page https://docs.nlx.io/try-nlx/management/getting-up-and-running we tell the user needs to clone the NLX git repository. This results in a untagged version of the `docker-compose.management.yml` file. That file contains Docker images with the tag latest.


## Problem

The problem with this is that the settings in `docker-compose.management.yml` can mismatch with what the `latest` Docker tag points to (at the moment of cloning and pulling).

Also the user needs to clone the whole code base for a single file to run a demo.


## Proposed solution

Use a dedicated git repository for this demo and version Docker tags instead of `latest`.

- [x] Move docker-compose.management.yml to new repo 'nlx-try-me'
- [x] Add DEX config
- [x] Create demo PKI
- [x] Alter 'getting up and running'  section of docs.nlx.io
- [x] Make sure it is clear at every step this is just meant to get to know NLX, NOT FOR PRODUCTION
- [x] Add renovate bot for auto updating of docker-compose
- [x] Add README with again clear warnings

Story points: 3.0
Always show request button on overview when I can request access

**As a** user
<br>**I want** to see the request button on the overview page
<br>**so** i can request access

**:desktop_computer: How to demonstrate / e2e test(s):**

1.

**:white\_check\_mark: Acceptance criteria**

1. when the access request already has a state, show the button over the text of that state

**:link: Reference links**

1.

Story points: 2.0
Remove support for disk configuration source

In https://gitlab.com/gitlab-org/gitlab/-/issues/217912 we introduced `domain-config-source` configuration flag to allow users to choose which domain source to use between `disk` and `api`.

In %"14.0" we plan to remove support for old disk configuration source.

The purpose of this issue is to remove `domain-source`, some code may be hard to remove in one MR, so this can be extracted to follow-up issues.

Omnibus will need to be updated to remove this flag

If users can't use the disk config source, we can gradually get rid of old code supporting it.

Story points: 2.0
Fix data race in gitlab source cache package

The following discussion from !246 should be addressed:

- [ ] @nolith started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/246#note_294638273):  (+1 comment)

    > Here the race is deep inside the package
    > 
    > we make use of global variables and this test will alter `retrievalTimeout` when a goroutine from the previous one is still running.
    > 
    > This will require extra effort to fix, but I think we should add the race detector now.
    > 
    > ```
    >  ==================
    >  WARNING: DATA RACE
    >  Write at 0x000000e0d4f0 by goroutine 46:
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.TestResolve.func9()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache_test.go:247 +0x3e
    >    testing.tRunner()
    >        /usr/local/go/src/testing/testing.go:827 +0x162
    >  Previous read at 0x000000e0d4f0 by goroutine 44:
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.(*Retriever).Retrieve()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/retriever.go:22 +0x8d
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.(*Entry).retrieveWithClient()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/entry.go:92 +0xc5
    >  Goroutine 46 (running) created at:
    >    testing.(*T).Run()
    >        /usr/local/go/src/testing/testing.go:878 +0x659
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.TestResolve()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache_test.go:246 +0x1d5
    >    testing.tRunner()
    >        /usr/local/go/src/testing/testing.go:827 +0x162
    >  Goroutine 44 (running) created at:
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.(*Entry).Retrieve.func1()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/entry.go:64 +0x67
    >    sync.(*Once).Do()
    >        /usr/local/go/src/sync/once.go:44 +0xde
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.(*Entry).Retrieve()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/entry.go:64 +0xad
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.(*Cache).Resolve()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache.go:89 +0x1ea
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.TestResolve.func8.1()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache_test.go:239 +0x76
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.withTestCache()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache_test.go:89 +0x185
    >    gitlab.com/gitlab-org/gitlab-pages/internal/source/gitlab/cache.TestResolve.func8()
    >        /builds/nolith/gitlab-pages/internal/source/gitlab/cache/cache_test.go:238 +0xc7
    >    testing.tRunner()
    >        /usr/local/go/src/testing/testing.go:827 +0x162
    >  ==================
    > ```

Story points: 1.0
Tech evaluation: Serve pages from object storage

From https://gitlab.com/gitlab-org/gitlab/issues/208135

The idea is to create proof of concept for pages to be able to serve content from object storage. This will hopefully help identify some of the unknowns to enable this functionality!

Will use [GitLab's object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/) with the [GDK config](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/object_storage.md) to explore a possible solution for this.

## Results

What we've discovered from !258

* Adding specific code to handle objects from AWS S3 or GCS may not be scalable and would be harder to maintain.
* Implementing [this diagram](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/258#note_310076437) it will be super slow, as Rails and Workhorse validate metadata multiple times, and is not very efficient if we want to random access the file (it does not cache anything).
* We should explore using presigned URLs #377

Story points: 1.0
Tech Evaluation: Support HSTS on GitLab Pages

## Problem to solve 
We need to research the technical implementation details to solve https://gitlab.com/gitlab-org/gitlab-pages/issues/28

## Proposal 
Spend 2-3 days assessing sorting capabilities and breakdown issue into smaller components if larger than 2 MRs: 
Some users want to enforce no access to their web content without HTTPS/certificates. This can be done with [HTTP Strict Transport Security (HSTS) policy](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security), but we do not currently support enabling this for GitLab Pages sites.

Story points: 1.0
GitLab Pages depends on the availability of GitLab API

If gitlab API is not available for some reason(e.g. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1936) GitLab Pages currently will become unavailable too(we clear cache in case of any API lookup problem).

If we not clear cache in these cases, GitLab Pages will be more independent and won't produce spikes of errors like https://dashboards.gitlab.net/d/web-pages-main/web-pages-overview?orgId=1&from=1586889900000&to=1586891700000&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-sigma=2

The following discussion from !194 should be addressed:

- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/merge_requests/194#note_255169959):  (+1 comment)

    > Should we consider replacing entry only if the new received one does not have `entry.lookup.Error` to solve intermediate errors of processing lookups?
    > 
    > I consider the today it can happen that we store the lookup of success request, but after refresh we receive error, like 500 from GitLab. It would help if we could catch with this mechanism a short living errors of GitLab API, and try to re-use success requests for as long as long interval. This reduces Pages dependence on API being super stable, and we anticipate that upstream API is flaky to some extent.
    > 
    > Or maybe better is in such case, is to re-use current lookup and extend the lease on the lookup to allow another refresh?
    > 
    > ```golang
    > if entry.response != nil && entry.response.Error != nil {
    >   entry.response = e.response
    > }
    > ```
    > 
    > > Note: I'm fine following that in next MR as an stability improvement.

Story points: 2.0
Make GitLab client timeout / JTW token expiry configurable

The following discussion from !201 should be addressed:

- [ ] @ayufan started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/merge_requests/201#note_251814520):  (+3 comments)

    > Is this timeout too small?
    > 
    > Can we make it larger, or at least configurable?

Story points: 2.0
XSSI in Gitlab Pages *.gitlab.io

**[HackerOne report #719770](https://hackerone.com/reports/719770)** by `ngalog` on 2019-10-22, assigned to @jeremymatos:

Security workflow issue: https://dev.gitlab.org/gitlab/gitlabhq/issues/2967.

### Summary  
Since now gitlab.io has implemented access control for all private pages, all pages should be limited to authorized user, however I found there is a way to steal content of JS files of victim using the XSSI(Cross Site Scripting Inclusion) technique in certain conditions.

### Steps to reproduce  
- Use Gitlab CI/CD to build a page, and limit access to project member only  
- In the gitlab page folder, upload a JS file, something like below

`secret.js`

```  
aws_key='AKIARONCHAN';  
aws_secret='RONCHANAGAIN'  
```  
- Then, as the victim, login to gitlab  
- As the attacker, in a domain that you control, prepare below html file and make victim to visit the page

https://attacker.com/attackpage

```  
<script src="https://GROUPID.gitlab.io/PROJECTID/secret.js"></script>  
<script>alert(aws_key);</script>  
```

- Then you should see a pop up containing the JS secrets of the gitlab page.


### Impact  
XSSI to steal user's gitlab page's JS file content

### Examples  
Given above

### What is the current *bug* behavior?  
Didn't block invalid refer

### What is the expected *correct* behavior?  
Should not be able to allow attacker to carry out XSSI attacker

### Relevant logs and/or screenshots  
![Screenshot_2019-10-22_at_9.46.24_PM.png](https://h1.sec.gitlab.net/a/bb6765a1-a5b7-4bff-b992-6a6e84553263/Screenshot_2019-10-22_at_9.46.24_PM.png)  
![Screenshot_2019-10-22_at_9.46.36_PM.png](https://h1.sec.gitlab.net/a/c7948fcf-c65c-449d-8f23-58add7b4b289/Screenshot_2019-10-22_at_9.46.36_PM.png)  
![Screenshot_2019-10-22_at_9.46.42_PM.png](https://h1.sec.gitlab.net/a/08c44e4a-8ac2-47ab-b817-71f750589783/Screenshot_2019-10-22_at_9.46.42_PM.png)

### Output of checks  
gitlab.com

## Impact

XSSI

## Attachments

**Warning:** Attachments received through HackerOne, please exercise caution!

* [Screenshot_2019-10-22_at_9.46.24_PM.png](https://h1.sec.gitlab.net/a/bb6765a1-a5b7-4bff-b992-6a6e84553263/Screenshot_2019-10-22_at_9.46.24_PM.png)
* [Screenshot_2019-10-22_at_9.46.36_PM.png](https://h1.sec.gitlab.net/a/c7948fcf-c65c-449d-8f23-58add7b4b289/Screenshot_2019-10-22_at_9.46.36_PM.png)
* [Screenshot_2019-10-22_at_9.46.42_PM.png](https://h1.sec.gitlab.net/a/08c44e4a-8ac2-47ab-b817-71f750589783/Screenshot_2019-10-22_at_9.46.42_PM.png)

Story points: 4.0
Default to JSON Logging

[We are moving to make JSON the default configured log format for GitLab Pages in 12.0 Omnibus release](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4102).

Would we like to make this the source install default as well by changing https://gitlab.com/gitlab-org/gitlab-pages/blob/master/logging.go#L28 ?

Story points: 1.0
Convert https variable to Gorilla ProxyHeaders

We temporary saved `https` flag in the context in https://gitlab.com/gitlab-org/gitlab-pages/merge_requests/168/diffs, but `https` variable is still being passed through a lot of calls.

We can use https://godoc.org/github.com/gorilla/handlers#ProxyHeaders instead of both these solutions.

We'll remove this flag in three steps 

- [x] Merge https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/225 to log an error if `https` flag does not match the `r.URL.Scheme`
- [x] Test bed with using a reverse proxy see https://gitlab.com/gitlab-org/gitlab-pages/issues/219#note_281592054 
- [x] Completely remove the `https` flag from the context

<details>
<summary>Old description</summary>

The following discussion from !141 should be addressed:

- [ ] @nick.thomas started a [discussion](https://gitlab.com/gitlab-org/gitlab-pages/merge_requests/141#note_167068450):  (+1 comment)

    > I know we use this imperative style of middleware (returning true/false to indicate whether to end) elsewhere in gitlab-pages, but I wonder if we could avoid it in this case, and instead use the `http.Handler` style instead?
    > 
    > One approach: the `Middleware` type would have a `ServeHTTP()` method and a `next` member. In the cases where we currently `return false`, we'd instead `m.next.ServeHTTP(w, r) ; return`. Otherwise, we'd serve the redirect and return. 
    > 
    > We use this pattern in workhorse and it serves us quite well. WDYT?

This is mostly a refactoring issue, I think  it would make the gitlab-pages codebase easier to understand, and we could make use of more third-party implementations this way.

One example: we could start using https://godoc.org/github.com/gorilla/handlers#ProxyHeaders on *just* the `ServeProxy` route!

</details>

Story points: 2.0
Custom error pages not served for namespace domains

Summary
===

`/404.html` not served even though it exists.

Steps to reproduce
===

Create a GitLab Pages website with `/404.html`. Go to `/non/existing_file`.

Expected
===

`/404.html` served.

Actual
===

Default GitLab 404 page served when using `*.gitlab.io`, but `/404.html` served when using custom domain.

Useful links
===

Public repository with a minimal, complete, verifiable example: https://gitlab.com/error-page-demo/error-page-demo.gitlab.io

Non-existent GitLab Page: https://error-page-demo.gitlab.io/non/existing_file


Page that should be served: https://error-page-demo.gitlab.io/404.html

Relevant documentation: https://docs.gitlab.com/ce/user/project/pages/introduction.html#custom-error-codes-pages

Story points: 2.0
Load domain data from API instead of traversing filesystem

We've seen a number of bugs where stale values of `config.json` are stored or stale directories mess up the domains loading. I think it might make sense to switch GitLab Pages to load the data from the API. We are doing this to support access control in !94.

- [ ] CE: Add API endpoint to serve project pages config.json
- [ ] Pages: Fetch and cache config when it does not exist

Story points: 5.0
Support HSTS on GitLab Pages

## Problem to Solve

Some users want to enforce no access to their web content without HTTPS/certificates. This can be done with [HTTP Strict Transport Security (HSTS) policy](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security), but we do not currently support enabling this for GitLab Pages sites.

## Proposal

By allowing for enabling this as a custom configuration/policy associated with a Pages site, we can make this possible.
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->

Story points: 1.0