giseldo/neodataset huggingface dataset:
Implement 404 page

**As a** _user_
<br>**I want** _goal_
<br>**so** _purpose_

**:desktop_computer: How to demonstrate**

1. Enter a non-existing url (eg: `<base_url>/asdf`) and see 404 page

**:white\_check\_mark: Acceptance criteria**

Depends on:
- page template

Todo:
- implement design
- show 404 page when a route is invalid / not found

**:link: Reference links**

1. https://sketch.cloud/s/eDOWa/a/PEzbMP

Story points: 1.0
Implement page template

**As a** _user_
<br>**I want** _goal_
<br>**so** _purpose_

**:desktop_computer: How to demonstrate**

1.

**:white\_check\_mark: Acceptance criteria**

Todo:
- fetch requirements from Peter about what sections should be scrollable / fixed
- user account info at top
- gemeente + environment info at top
- sidebar including only the link 'NLX Management' (links to root), 

**:link: Reference links**

1. https://sketch.cloud/s/eDOWa/a/rPQ5lD

Story points: 5.0
Implement authentication mechanism

**As a** system administrator
<br>**I want** login to management using open-id connect
<br>**so** securely manage NLX

**:desktop_computer: How to demonstrate**

1. in style

**:white\_check\_mark: Acceptance criteria**

1. remove username password
1. add open-id connect
1. docker-compose example with dex
1. developer documentation, no user documentation yet

**:link: Reference links**

Use appstore history

1. https://sketch.cloud/s/eDOWa/a/7L27xY

Story points: 5.0
Implement E2E test tooling

**As a** _user_
<br>**I want** _goal_
<br>**so** _purpose_

**:desktop_computer: How to demonstrate**

1.

**:white\_check\_mark: Acceptance criteria**

- E2E-test setup using BrowserStack and initial test (see setup Appstore)
- a11y-test setup using axe + test welcome page (see setup DON)

**:link: Reference links**

1.


- can we run an actual instance of the management-ui as part of a Review App? Can we give BrowserStack access to it? -> yes, in review app

Story points: 3.0
Implement Welcome page

**As a** _user_
<br>**I want** _goal_
<br>**so** _purpose_

**:desktop_computer: How to demonstrate**

1.

**:white\_check\_mark: Acceptance criteria**

- setup CI/CD
- setup fresh CRA in 'nlx-management'
- implement welcome page
- path: /login
- out of scope: login button, should be done as part of the dex story
- create separate `@commonground/linting package` with our shared linting configuration. As a base [this repo](https://github.com/publicJorn/eslint-config-jorn) can be used. It's a copy of react-scripts. No specific work was done for it yet.  (we'll base it on standardjs) **@publicJorn**

**:link: Reference links**

1. https://sketch.cloud/s/eDOWa/a/7L27xY

Story points: 8.0
Extract components from Appstore to CG Design System

**As a** _user_
<br>**I want** _goal_
<br>**so** _purpose_

**:desktop_computer: How to demonstrate**

1.

**:white\_check\_mark: Acceptance criteria**

- remove components from CG DS which we don't use in DON and Appstore
- provide default theme.js
- add global styling, Typography and Button components
- take Peter his thoughts into account (https://sketch.cloud/s/D9YpW)
- update [DON](https://gitlab.com/commonground/developer.overheid.nl/-/issues/194) & [Appstore](https://gitlab.com/commonground/appstore/appstore/-/issues/170) so they use these components


**:link: Reference links**

1. https://gitlab.com/commonground/appstore/appstore/issues/165

**:hammer_and_wrench: Technical solution:**

* [x] Chart what components are currently used in our apps
* [x] Remove all unused components
* [x] Update dependencies
* Add the following anew
  * [x] Global styling (extendable)
  * [x] Theming (extendable)
  * [x] Typography (Discuss: in global styling or as components?)
  * [ ] ~~Grid~~
  * [x] Button
* [x] Create tests with [RTL](https://testing-library.com/docs/react-testing-library/)

Story points: 8.0
Add abstraction layer (txlog-api) between inway/outway and txlog-db

Currently inways and outways directly write via a Postgres interface to the txlog-db. It would be great to introduce a layer of abstraction here (txlog-api), so the database is more pluggable and security is enhanced.

Currently, only `ListRecords` is implemented.

:bangbang: This is a breaking change, since you need to specify an extra flag with the TXLog API address to the Inway & Outway. We can make this backwards compatible by supporting both a `TXLOG API` address and a `TXLOG Postgres DSN`. The current issue assumes we **WILL** make this backwards compatible.

**Todo**

- implement write call to create a single record
- update Outway and Inway
- support using both the txlog Postgres DSN flag and the txlog API flag
- attempt to use CQRS pattern?

Story points: 8.0
Display helpful error message when loading wrong certificate

### User story

*For components that load certificates: inway, outway, directory*

As a **developer**
<br>I want to **see why the provided certificate is not valid**
<br>so that **I know what the problem is**

### How to demonstrate
* Load certificate that is not signed by the provided root cert
* Should display, that the cert is only valid for '$ca_1', and not for provided '$ca_2'


**:hammer\_and\_wrench: Proposed technical solution**

* Read information about provided certificates and display it

Story points: 2.0
Add support for intermediate certificates in chain

As a **user**
<br>I want to **use a certificate that is signed by an intermediate certificate**
<br>so that **I can use PKI Overheid certificates**

---

Problem right now is that the eg. the inway only offers the leaf certificate in the TLS handshake. The other party, eg. the directory, can verify that certificate because the intermediate certificates are missing. The directory only has the root certificate.

**:movie_camera: How to demonstrate**
* -
* -

**:mag_right:(Test) scenarios**<br>
*Title*<BR>
Given demo env
When using cert without intermdiate
Then it should keep working

**:white\_check\_mark: Acceptance criteria**

1. Update NLX documentation about [going prod](https://docs.nlx.io/use-nlx/request-a-production-cert)
1. Unit Tests
1. NLX works with intermediate certificates between all NLX components


**:hammer\_and\_wrench: Proposed technical solution**

* Load all the certificates, including the intermediate, and use them during TLS handshake
* Use full chain cert files
* See: !1207

**:link: Reference links**

* 


Story points: 5.0
More granular authorization on inways using certificate pinning

### User story

As an **ops person**
<br>I want to **be able to authorize more granularly on inways**
<br>so that **I can improve the security of the NLX system in general**

NLX currently offers a way to whitelist organizations based on an organization name. This is really easy, but also introduces a large attack vector as every trusted certificate that is handed out for an organization can automatically be used in the context of NLX.

To allow more granular authorizations, we would like to introduce authorizations based on the `subjectPublicKeyInfo` field in the certificate. This has the following benefits:

- This restricts the certificates from a particular organization that can be used to setup NLX connections.
- This allows organizations to compartiment their security by having different security zones.
- This protects the NLX system of compromised CA's by pinning to specific public keys of certificates.

Of course this results in more administrative work by the inway administrator, but we can probably solve this partially by implementing good UX confirmation flows as described in #742. This could be a great follow-up story.

**:white\_check\_mark: Acceptance criteria**

1. The service-config.toml and config API are extended to support authorization based on public keys
1. We describe authorization based on the public key as the preferred way of authorization in production in the docs
1. When multiple authorization options are set, they all need to match before authorization is confirmed.
1. Remain backwards compatible and announce deprecation for the old way of authorizing in the service-config.toml.
1. Documentation about this feature is added to the docusaurus

**:wrench: Proposal**

*service-config.toml*

```toml
[services.postman-echo]
endpoint-url = "https://postman-echo.com/"
documentation-url = "https://docs.postman-echo.com/"
authorization-model = "whitelist"

[[services.postman-echo.authorization-whitelist]]
organization-name = "Gemeente Utrecht"
public-key = "sha256:GX6v14r7KP+TTQPzzmg3XgcaEVfcsCgUx6TfDCHTu5M="

[[services.postman-echo.authorization-whitelist]]
public-key = "sha256:/YAbDkjkJ6/2961u8Mun+Y8RVygDbSSoH5ZiSSdeCsU=" # Gemeente Amsterdam

[[services.postman-echo.authorization-whitelist]]
organization-name = "Gemeente Haarlem"
```

We could for example use a SHA-256 hash of the public key, similar to the [HTTP Public Key Pinning](https://developer.mozilla.org/en-US/docs/Web/HTTP/Public_Key_Pinning) (HPKP) header (thanks @hugoboos).

Example similar to HPKP:

```bash
openssl x509 -in org.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64
GX6v14r7KP+TTQPzzmg3XgcaEVfcsCgUx6TfDCHTu5M=
```

Or skip the base64 step:

```bash
openssl x509 -in org.crt -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256
197eafd78afb28ff934d03f3ce68375e071a1157dcb02814c7a4df0c21d3bb93
```


Story points: 5.0
Restructure Helm charts and upgrade Helm to v3

As a **Developer**
<br>I want to **deploy using Helm v3**
<br>so that **I can use up to date tooling**

**:white\_check\_mark: Acceptance criteria**

1. The charts can be used with Helm 3
1. The charts can be used by ourselves for development / testing purposes, but also by users
1. The charts are structured per (set of) component(s)
1. It is possible to install multiple releases in the same Kubernetes namespace
1. Documentation is updated accordingly
1. A review app is limited to one Kubernetes namespace

**:pencil2: Proposed structure**

*Central*
- directory
- certportal
- docs

*Organization*
- inway
- outway
- insight
- management

Reference information
- Check out Helm repository of signals application



## Tasks

### Update to Helm v3

* [x]  Update deployment Docker image
* [x]  Update deployments
  * [x]  NLX deployment
  * [x]  AppsStore deployment
  * [x]  DON deployment
* [x]  Migrate clusters
  * [x]  Review
    * [x]  Migrate releases
    * [x]  Remove Tiller
  * [x]  Delta
    * [x]  Migrate releases
      * [x]  cgn-prod
      * [x]  go-nlx-io-prod
      * [x]  hackathon-services
      * [x]  nlx-demo
      * [x]  nlx-acc
      * [x]  nlx-preprod-directory
      * [x]  nlx-prod-directory
      * [x]  nlx-io-prod
      * [x]  nlx-prod-demo
    * [x]  Remove Tiller


### Restructure charts

#### Core

- [x]  Directory UI
- [x]  Directory
- [x]  Directory monitor
- [x]  Docs


#### Organisation

- [x]  Inway
- [x]  Outway
- [x]  Unsafe CA
- [x]  Config API
- [x]  Transaction log
- [x]  Auth service
- [x]  NLX Management 


#### Demo

- [x]  Insight API
- [x]  Insight UI
- [x]  Cert portal
- [x]  Demo application
- [x]  IRMA server


#### Support

- [x]  Review page

Story points: 8.0
Migrate docs.nlx.io to Docusaurus 2

As a developer
<br>I want to use only one documentation tool
<br>so that I can fix bugs only once

### How to demonstrate
* Show a screenshot of the previous version
* Navigate to the new version
* Show that they look mostly the same

### (Test) scenarios
*Same look and feel*  
Given the documentation site  
When I visit it  
Then looks and feels similar to the previous edition

*No external CSS and JS*  
Given the documentation site  
When it is loaded in the browser  
Then it loads only resources from the documentation site itself and algolia

**:white\_check\_mark: Acceptance criteria**

1. docs.nlx.io is served using Docusaurus 2
1. It has the same look and feel as the Docusaurus 1 implementation
1. No CCS and JS files are loaded from external CDNs
1. CSP Rules allow for no external css or js files

**:hammer\_and\_wrench: Proposed technical solution**

* Follow the [migration documentation](https://docusaurus-2.netlify.com/docs/migrating-from-v1-to-v2/) for Docusaurus 2 


Story points: 2.0
Display friendly error pages

### User story

As a **user**
<br>I want to **see a nice error page instead of the default nginx error page**
<br>so that **I know what is wrong and have options to further navigate the websie**

The [documentation website](https://docs.nlx.io) shows the default nginx error pages in case of an http error:

![Screen_Shot_2019-11-14_at_11.16.00](/uploads/b142c95e8259b4f159ec1a840b466b91/Screen_Shot_2019-11-14_at_11.16.00.png)



**:white\_check\_mark: Acceptance criteria**

1. add 404 custom page containing a user friendly error message
1. all 404 errors must redirect to this page
1. custom page must include NLX logo and a link back to docs.nlx.io root
1. the scope of this page is docs.nlx.io


**:hammer\_and\_wrench: Proposed technical solution**

* create new page in docusaurus which is not present in navigation

**:link: Reference links**

*

Story points: 2.0
Alter TTL for directory entries to 86.400 seconds

As a **maintainer**
<br>I want to **clean up the directory by removing off-line services**
<br>so that **the directory content stays relevant**

### How to demonstrate
*
*

### (Test) scenarios
*Title*<BR>
Given <BR>
When <BR>
Then <BR>


**:white\_check\_mark: Acceptance criteria**

1. Directory regularly cleans up offline services after being unreachable for set Time To Live (in seconds)
1. Set Time To Live to 1 day (86.400 seconds)
1. Documentation **(!)**
1. Tests

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

* 


Story points: 1.0
Display in- and outway versions in dashboard

**As a** maintainer
<br>**I want to** see which versions of the inway & outway are being used
<br>**so that** I can make better decisions about release management and communicate about them

---

### (Test) scenarios
*Title*<BR>
Given the stats api and version info available in the database<BR>
When I open the grafana dashboard<BR>
Then I see one line graph per environment with a counter of inways and on for the outways per version <BR>


**:white\_check\_mark: Acceptance criteria**

1. The NLX Grafana dashboard should show a graph with the version of the inways and outways being used.
1. One line per version representing usage over time (x axis = time, y axis = count)
1. Display a graph for the inways and one for the outways. Show both graphs for all environments (DEMO, pre-PROD & PROD).

For inspiration:

![Screenshot_2019-11-05_at_10.40.37](/uploads/f41793c35b7fe55a32ea80605240c571/Screenshot_2019-11-05_at_10.40.37.png)

**:hammer\_and\_wrench: Proposed technical solution**

* use /stats endpoint to the inspection api to pull data to display/mine OR use influx idea of Hugo 

**:link: Reference links**

*

Story points: 3.0
Store inway and outway version in use per environment

### User story

**As a** maintainer
<br>**I want to** store the versions of inways and outways per environment
<br>**so that** I can make better decisions about release management and communicate about them

---

### (Test) scenarios
*Title*<BR>
Given an inway<BR>
When the inway registers itself<BR>
Then the inway will send the version to the directory and that is stored in the database<BR>

Given an outway<BR>
When the outway fetches the inways from the directory<BR>
Then the outway will send the version to the directory and that is stored in the database<BR>

Given the /stats endpoint  
When the /stats endpoint is requested  
Then the raw data is returned to the client

**:white\_check\_mark: Acceptance criteria**

1. all inways and outways in demo, pre-prod and prod are added
1. there must be an endpoint which is used to get the version info


**:hammer\_and\_wrench: Proposed technical solution**

* for the inway we move the inspection of the version from the monitor to the registration
* for the outway we add it as parameter in the request to the directory
* we store both versions in the directory for the inways, we add the version of the outways in a new table
* add /stats endpoint to the inspection api. use this endpoint to add stats to the database and pull data to display/mine OR use influx idea

**:link: Reference links**

* 



Story points: 5.0
Notify system administrators of updates and breaking changes


***Needs a new refinements since this will depend on #867***<br>
New:
* Generate notifications as all notifications in new NLX Management
* Allow users to configure how they want to be notified

---

**As a** system administrator
<br>**I want to** know when I need to update NLX and when breaking changes are introduced
<br>**so that** I can stay secure and up-to-date

**:white\_check\_mark: Acceptance criteria**
 
1. receivers are technical email addresses of inways
1. Is part of the pipeline, we send out a notification when a new release is made
1. must be a distinct list of email addresses (distinct over the three environments - demo, pre-prod and prod)
1. support@nlx.io is the from address
1. fixed email content with a variable latest version inclusive a remark how to unsubscribe
1. add unsubscribed mail addresses table (not in a public repro) which is used to remove those addresses from the mailinglist


**:hammer\_and\_wrench: Proposed technical solution**

* Implement mail job which fetches all inway email addresses and mails a template mail where version and change log link is added.
* auto fill mailing list with email addresses of inways (in the directory)
* its a new component within our cluster
* When we hand out certificates for the demo, pre-prod and prod environment we register one or more contacts

**Not in this story**
* addresses of outways
* for future reference mayne: subscription page, link to this page in the documentation in docusaurus

Story points: 8.0
Add Review App URL overview page

**As a** reviewer and/or approver
<br>**I want to** be able to use an overview page with all relevant Review App urls
<br>**so that** I don't have to search for them in the deployment job console logs

---

Right now it's not easy to find the URL's necessary to review and / or approve the Review App deployment. Those are generated by Review App and can be found in de deployment job log. 

**:white\_check\_mark: Acceptance criteria**

1. Create a release specific page with all Review App URL's that are served by Ingress
1. Deploy this page only in case of a Review App deployment
1. Starting a Review App from Merge Request page opens this new page in browser.
1. Don't spend any time on layout, just the URL's in order of deployment will do

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
* 

Story points: 3.0
Optimize build times by parallelisation and caching

**As a** maintainer
<br>**I want to** see my builds succeed quickly
<br>**so that** I can focus on coding

**:white\_check\_mark: Acceptance criteria**

1. Builds of dependencies are done in parallel
1. Builds of code that does not change are available from cache
1. Significant reduction of build times

**:hammer\_and\_wrench: Proposed technical solution**

* Use Kaniko https://skaffold.dev/docs/how-tos/builders/#dockerfile-in-cluster-with-kaniko
* Check if all Docker layers are cached correctly


**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)

Story points: 5.0
Don't strip data-subjects in outway

In #694 we discovered that outways strip data-subject headers from calls. This means that the inway cannot log this header, resulting in inability to disclose logs via Insight API.

When fixing this, update the header docs as well.


*Expected behaviour*

* data-subject header should not be stripped

*Actual behaviour*

* data subject is stripped by outway

**:clipboard: Steps to reproduce**

1. (reproducibility is very important)
1.
1.


**:beetle: Cause**

* (if cause is known, details here)
* (if you can pinpoint a piece of code, link to line number)


**:mag: Details**

(Paste any relevant logs - please use code blocks to format console output, logs, and code)

(Add screenshots using `Attach a file`)

---

Story points: 1.0
Replace Hugo docs with Docusaurus

We'd like to improve our documentation as part of #505.

- replace hugo docs with Docusaurus (Helm charts, Docker images, Skaffold...)
- use documentation files as is in Docusaurus. we can cleanup/style in a next iteration
- add current content as part of the English language
- marketing pagina stays at is
- use default Docusaurus, but use the NLX colors
- keep the current links in the navbar
- tested in Chrome, Firefox and Safari

**For further reference**
- Can and how will we toggle between Linux, Mac OS & Windows environments?
See https://facebook.github.io/react-native/docs/getting-started

Story points: 8.0
Dashboard with NLX in numbers

**As a** maintainer
<br>**I want to** know how many organisations & services are active on the provided environments
<br>**so that** I have a measure of success and growth

**:white\_check\_mark: Acceptance criteria**

1. Grafana dashboard with graphs per indicator:
* Cumulative number of organisations over time 
* Cumulative number of services over time 
1. Above graphs for every provided environment (test, accept, preprod, prod, demo)
1. Interval is every 24-hours at 06:00
1. Above graphs all with x-axis covering the same date range: from date $first_organisation_in_production_environment until $today
1. Backups will be done as part of #777

---
Example 
![Screenshot_2019-10-03_at_18.22.04](/uploads/fbedacf3a0699ba46f3d3d4b53cb6c5a/Screenshot_2019-10-03_at_18.22.04.png)
1.

**:hammer\_and\_wrench: Proposed technical solution**

* Fetch data using a cron job in Grafana / Influx DB node
* Fetch data using the public Directory API endpoints (/list-organizations, /list-services)

**:link: Reference links**

* https://directory.demo.nlx.io/api/directory/list-organizations
* https://directory.demo.nlx.io/api/directory/list-services

* [Project Definition of Done](https://gitlab.com/commonground/nlx/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
* [Ansible roles for the benchmark cluster](https://gitlab.com/commonground/nlx/benchmark)

Story points: 3.0
Extend demo environment

**As a** potential user
<br>**I want to** see the most important concepts of NLX in action
<br>**so that** I can decide if I want to use NLX

---

Current demo shows Insight API concept with IRMA. We need more demo's, to be able to show every relevant concept in a permanent online environment.

For this, we extend the current demo UI with a navigation structure to browse through several demo's. Each demo 

Let's start with an overview page that, to start with, contains just the insight demo.

**:white\_check\_mark: Acceptance criteria**

1. Permanent online Demo landing page explaining NLX at demo.nlx.io
1. Landing page offers the insight demo as an option, with room to expand
1. In a separate demo repository: https://gitlab.com/commonground/nlx/demo
1. Hosted in its own namespace, but using API's from the demo environment (demo's run with demo certs)


**:hammer\_and\_wrench: Proposed technical solution**

* A separate Helm chart with the name nlx-demo that provides all demo-related components
* This Helm chart is automatically deployed through `.gitlab-ci.yml`
* There are no separate environments for nlx-demo, only one production environment


Story points: 8.0