giseldo/neodataset huggingface dataset:
Add authentication to management-api

**As an** administrator
<br>**I want to** have a secure management-api
<br>**so that** no unauthenticated people can make changes to the API

**:white\_check\_mark: Acceptance criteria**

1. Every API endpoint (except /login) is secured and will check if the user is authenticated
1. There should be a login screen as part of the UI
1. There should be an option to logout from the UI
1. Only support the roles 'admin', 'readonly' for now
1. When not authenticated, the UI should always redirect to the login screen
1. The authenticatie mechanism should be pluggable. Looking up logins from a plain text file is fine for now.
1. Token should expire after 60 minutes (don't implement refresh logic for now)

**:hammer\_and\_wrench: Proposed technical solution**

1. Add & implement a /login endpoint
1. It is possible to specify multiple users, passwords (hashed) and a role in a plain text file

# LATER
1. When logged in, user name should be shown top right in Management UI.
1. When clicking on user name, there should be option to log out


*The overall idea is that one Management Dashboard gives infra admins the possibility to manage their NLX components (while not being able to access functional logs) and gives privacy auditors and / or functional managers the ability to look up transaction logs (while not being able to alter the NLX network config)*


Story points: 8.0
Create initial management-ui

**As a** administrator
<br>**I want to** administer my NLX components from a fancy UI
<br>**so that** I do not have to use the CLI anymore

**:white\_check\_mark: Acceptance criteria**

1. A new React app is setup with create-react-app in `management-ui`
1. An initial Router setup is done with react-router with an inway page and service page
1. The system shows a list of Inways and Services
1. It is possible to create a new service, update an existing service and delete a service

Story points: 13.0
Document header specifications

**As a** end-user developer
<br>**I want to** know the specifications of the NLX http headers
<br>**so that** I can design my datamodel to store the data.

**:white\_check\_mark: Acceptance criteria**

1. Document NLX header specifications (type, size, optional or not)
1. Add this as content on docs.nlx.io including navigation
1. Link to this page frrom relevant sections

**:hammer\_and\_wrench: Proposed technical solution**

* n/a

**:link: Reference links**

* [Header documentation currently](https://docs.nlx.io/further-reading/transaction-logs/)

Story points: 2.0
Make insight-api and irma-api-server configurable and disable on the minikube by default

**As a** developer
<br>**I want to** have a fast developer environment
<br>**so that** I can develop quickly

**:white\_check\_mark: Acceptance criteria**

1. The services insight-api and irma-api-server are configurable in the organization Helm chart
1. These services are disabled by default on the minikube environment

Story points: 1.0
Create a management API to manage inway and service configuration from UI

**As a** developer
<br>**I want to** by able to manage the inway and service configuration through the management-api
<br>**so that** we can start building the management-ui

**:white\_check\_mark: Acceptance criteria**

1. A new service `management-api` is added to the codebase
1. It is possible to PUT new inway configurations to the `management-api`
1. These inway configuration updates are persisted in etcd
1. There is no authentication or authorisation performed yet on this service (later enhancement)


**Technical implementation**

No gRPC, use REST/JSON

Story points: 8.0
Enhance the quality of NLX error messages

**As a** user 
<br>**I want** error messages that tell me exactly what is going on
<br>**so that** I don't have to waste time analyzing

**:white\_check\_mark: Acceptance criteria**

1. Possible errors are identified and no longer ambiguous
1. Errors during setup / usage inway and outway are clear
1. Possible errors are documented with a short explanation 
1. +kudos if error message links to docs

**:hammer\_and\_wrench: Proposed technical solution**

1) identifield possible failures.

client -> outway -> F1 -> inway -> F2 -> service.

directory health check -> F3 inway.

directory list 'unhealty' services.

Cases F1, F2 now produces clear logs.
Case F3 now no longer is cause for removal from directory.

*

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
*

Story points: 8.0
Build the config API

**As a** user
<br>**I want to** have configuration API
<br>**so that** I don't need to start all xways with complex cli commands

--- 

**:white\_check\_mark: Acceptance criteria**

1. Configuration API with persistent storage backend
1. Configurations can be saved and retrieved
1. Configuration template for an inway
1. Configuration is validated by the API

**:hammer\_and\_wrench: Proposed technical solution**

* etcd as persistent storage
* communication between API and component GRPC

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
*

Story points: 13.0
Zap logger failed to sync

When stopping an outway or inway with ctrl+c, you see the following message:

```
failed to sync zap logger: sync /dev/stderr: invalid argument
```

*Expected behaviour*

* The Zap logger is successfully synced and no error is shown.

*Actual behaviour*

* Failed to sync the logger and displaying an error.

**:clipboard: Steps to reproduce**

1. Start outway (or inway) from console
1. Stop outway (or inway) with ctrl+c


**:beetle: Cause**

Not yet known

**:mag: Details**

N/A

---

Story points: 2.0
Enable HTTP2  between outway and inway

**As a** customer
<br>**I want to** have HTTP2 enabled between the outway and inway
<br>**so that** we use less CPU and gain an higher throughput

**:white\_check\_mark: Acceptance criteria**

1. Running an outway with `GODEBUG=http2debug=1` displays the usage of http2

**:hammer\_and\_wrench: Proposed technical solution**

*  Enable http2 transport in the outway

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)

Story points: 1.0
Switch back to Gitlab Gosec image for SAST

The following discussion from !657 should be addressed:

- [x] @RonaldKoster started a [discussion](https://gitlab.com/commonground/nlx/merge_requests/657#note_191689116):  (+1 comment)

    > `registry.gitlab.com/bartjkdp/gosec:feature-upgrade-to-gosec-2` image in personal registry? I would prefer something like`registry.gitlab.com/nlx/gosec:feature-upgrade-to-gosec-2`

Switch back to Gitlab image when MR is accepted.

Story points: 1.0
Directory inspection API get service api spec database error

The `/directory/get-service-api-spec` endpoint on the directory-inspection-api returns `Database error.`.

*Expected behaviour*

*  `200 Ok` response with the service api spec or `404 Not Found` when no spec exists.

*Actual behaviour*

*  500 Internel Server error response: `{"error":"Database error.","message":"Database error.","code":13}`

**:clipboard: Steps to reproduce**

1. GET request to https://directory.demo.nlx.io/api/directory/get-service-api-spec/convenantgemeenten/AppointmentDataService


**:beetle: Cause**


* Error is caused by `Scan` that returns an error. See https://gitlab.com/commonground/nlx/blob/master/directory-inspection-api/inspectionservice/get-service-api-spec-doc.go#L63


**:mag: Details**

*  When there are no inways registered:
```
inspectionservice/get-service-api-spec-doc.go:65	failed to execute stmtSelectServiceInway	{"version": "v0.73.0", "handler": "list-services", "error": "sql: no rows in result set"}
```

*  When there is a inway registered:
```
inspectionservice/get-service-api-spec-doc.go:65	failed to execute stmtSelectServiceInway	{"version": "v0.73.0", "handler": "list-services", "error": "sql: Scan error on column index 1, name \"api_specification_type\": unsupported Scan, storing driver.Value type <nil> into type *string"}
```

---

Story points: 3.0
Implement 503 HTTP status code for offline services

**As a** developer
<br>**I want to** use nlx and have sane error messages
<br>**so that** I can debug my new inway / outway

```
$ curl http://outway:9090/nlx/afval/
nlx outway: invalid path in url
```

Above error could have **many** causes which can be explained. A most of all the *PATH IS CORRECT.*

Communication failure between Inway and directory means no outway will be able to access the service. This makes nlx quite fragile.

Possible causes:

* Inway can't connect to upstream API: `[inway] http: proxy error: dial tcp 127.0.0.1:3000: connect: connection refused`


**:white\_check\_mark: Acceptance criteria**

1. A temporarily unavailable service should give back 503. Not a 4xx. since the user is not at fault.<br>
2. Make Outway work even when communication between inway and directory fails.

**:hammer\_and\_wrench: Proposed technical solution**

* The outway should also have knowlegde of offline services from the directory service.

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)

Story points: 1.0
Add first version of configuration API

**As a** user
<br>**I want to** configure and manage inways and outways from a central point
<br>**so that** I don't need to start all xways with complex cli commands

--- 

Out of scope (will be done later): 
* Keyless

**:white\_check\_mark: Acceptance criteria**

1. xway registration flow implemented 
1. xway self init and registration
1. management API for self init acknowledgement
1. Configuration API with persistent storage backend
1. xway watches configuration for changes
1. toml's are gone


**:hammer\_and\_wrench: Proposed technical solution**

* 

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
*

Story points: 20.0
Move scheduled security scanning to nlx-security repo

**As a** developer
<br>**I want to** move scheduled security scan to a seperate repo
<br>**so that** it doesn't interfere with the dev pipelines

---

Moving it to a separate repo allows us to:

* Reserve the gitlab-ci.yml for development pipelines, keeping it clean

**:white\_check\_mark: Acceptance criteria**

* [x]  Move scheduled security scan pipeline to repo nlx-security
* [x]  Security scans run on demo, preprod and prod
   * [x]  Dependency scanning
   * [x]  Static application security testing (SAST)
   * [x]  Container scanning
   * [x]  Dynamic application security testing (DAST)
* [x]  Report findings in the security dashboard, building a consistent history
* [x]  Add security scans to the nlx repo development pipelines that do NOT report to security dashboard
   * [x]  Dependency scanning
   * [x]  Static application security testing (SAST)

Story points: 3.0
Changes in inway configuration do not propagate to directory

When attributes of services change (e.g. api specification url or documentation url) these changes are not propagated to the directory. This can be solved by manually deleting the service from the PostgreSQL database. A new service is created with the correct information.

*Expected behaviour*

* When a change occurs in the inway configuration of a service, this is automatically propagated through tot the directory.

*Actual behaviour*

* The information is not updated in the directory.

**:clipboard: Steps to reproduce**

1. Start a service without an api spec URL on an inway and register to the directory
1. Add an API spec URL
1. See the spec URL is not visible within the directory.


**:beetle: Cause**

* (if cause is known, details here)
* (if you can pinpoint a piece of code, link to line number)


**:mag: Details**

Regression, worked before

(Paste any relevant logs - please use code blocks to format console output, logs, and code)

(Add screenshots using `Attach a file`)

---

Story points: 2.0
Directory: add a link to the documentation of an API

**As a** user
<br>**I want to** , when using the directory-ui, navigate to the documentation-url provided by the provider of an API
<br>**so that** I can read the documentation of an API 

**:white\_check\_mark: Acceptance criteria**

1. documentation-url link is available on the drawer of the Directory UI

**:hammer\_and\_wrench: Proposed technical solution**

* Display value of property `documentation_url` if present.

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
*

**:camera_with_flash: Screenshot**

![Screenshot_2020-12-08_at_13.32.51](/uploads/34b1198fafc76b8cafa350831012ebb8/Screenshot_2020-12-08_at_13.32.51.png)

Story points: 1.0
Enable gitlab-ci on nlx-demo repository

**As a** developer
<br>**I want to** automatically release Docker images for nlx-demo
<br>**so that** I do not have to do this manually

**Background**

When we moved from Jenkins to Gitlab CI we forgot to create a new configuration for the nlx-demo repo.

**:white\_check\_mark: Acceptance criteria**

1. Gitlab CI automatically Docker images when commits land on master

Story points: 1.0
Links in Docs don't work

In the Getting Started guide, links intended to lead from page to page are broken.
Requesting pages without trailing slash does not work

*Expected behaviour*

Links lead to a page

* consume-an-api.md: `[part 2](../create-certificates)` & `[part 4](../provide-an-api)`
* create-certificates.md: `[part 1](../setup-your-environment)`
* provide-an-api.md: `[part 2](../create-certificates)` & `[part 2](../create-certificates)`

*Actual behaviour*

Every link without trailing slash fails. Instead it attempts to load something from port 8080.

**:clipboard: Steps to reproduce**

1. Visit https://docs.nlx.io/get-started/setup-your-environment/
1. Follow the guide
1. Click every link in the guide

It's not just the guide but everything on docs

**:beetle: Cause**

* Might be an Nginx configuration error


**:mag: Details**


---

Story points: 2.0
Incorrect port for "Consume an API" in docs

Update incorrect port in docs.

*Expected behaviour*

* “You will get back the container id of the container you created from this image. By running this command, we’ve launched our very own NLX outway. It is running on http://localhost:8080.”

*Actual behaviour*

* “You will get back the container id of the container you created from this image. By running this command, we’ve launched our very own NLX outway. It is running on http://localhost:4080.”

**:clipboard: Steps to reproduce**

1. Go to docs "consume an API" page: https://docs.acc.nlx.io/get-started/consume-an-api/
1. Navigate to section "Setting up the outway"


**:beetle: Cause**

* (if cause is known, details here)
* (if you can pinpoint a piece of code, link to line number)


**:mag: Details**

(Paste any relevant logs - please use code blocks to format console output, logs, and code)

(Add screenshots using `Attach a file`)

---

Story points: 1.0
insight-api unit tests

**As a** developer
<br>**I want to** have unit tests on the insight-api
<br>**so that** I am kinda sure everything is working 

**:white\_check\_mark: Acceptance criteria**

1. Test what can be tested on the insight-api

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
*

Story points: 2.0
Eliminate txlog-ui and txlog-api

**As a** developer
<br>**I don't want to** maintain a deprecated application
<br>**so that** I can spend my energy on more useful tasks

TX Log UI is unsecure technical debt and should be eliminated.
Will be replaced later with UI in Management Dashboard.

**:white\_check\_mark: Acceptance criteria**

1.  update the docs - remove txlog UI and show Postgres guideline to show logs
1.  txlog-ui directory removed from repository
1.  txlog-api directory removed from repository
1.  pipelines updated
1.  helm / skaffold updated
1.  remove from Uptime Robot
1.  update nlx-compose repo


**:hammer\_and\_wrench: Proposed technical solution**

*  Copy Postgres query from txlog-api for guideline

**:link: Reference links**

* [Project Definition of Done](https://gitlab.com/commonground/nlx/blob/master/CONTRIBUTING.md#4-2-definition-of-done)
* 

Story points: 3.0
Add different issue templates

**As a** user
<br>**I want to** submit feature requests, bugs, support questions, etc.
<br>**so that** 

**:white\_check\_mark: Acceptance criteria**

1.  Issue template for User story
1.  Issue template for Bug
1.  Issue template for Support tickets
1.  Issue template for Security reports
1.  Template for Merge requests


**:hammer\_and\_wrench: Proposed technical solution**

*  https://gitlab.com/help/user/project/description_templates.md#using-the-templates

**:link: Reference links**

*

Story points: 1.0
Reduce noise from Renovate bot

**As a** developer
<br>**I want** to reduce the number of Pull Requests generated by Renovate Bot
<br>**so that** runners are available when I need them and I don't have to review and merge so many PR's every day

---

Renovate bot is great but generates too many PR's to comfortably process.
Let's reduce the noise.

**:white\_check\_mark: Acceptance criteria**

1.  Groups of dependencies are grouped into aggregated Pull Requests
1.  Renovate Bot only generates new PR's between 2:00h AM CET and 8:00 AM CET to keep runners available when devs need them

**:hammer\_and\_wrench: Proposed technical solution**

*   Enable Package Grouping: https://renovatebot.com/docs/noise-reduction/#package-grouping
*   Limit Renovate to the night with scheduling: https://renovatebot.com/docs/noise-reduction/#scheduling-renovate

**:link: Reference links**

* https://renovatebot.com/docs/noise-reduction/

Story points: 2.0
Remove FIELDLAB-WALKTROUGH.md

**As a** PO
<br>**I want to** remove the old fieldlab walktrough
<br>**so that** people don't start a guide that isn't accurate anymore

**:white\_check\_mark: Acceptance criteria**

1.  File FIELDLAB-WALKTROUGH.md is removed from repo root

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

*

Story points: 0.0
Rename NLX Stelsel to NLX System

**As a** PO
<br>**I want** the Readme.md to be in English
<br>**so that** everyone can understand it

**:white\_check\_mark: Acceptance criteria**

1.  Change title in Readme.md from NLX Stelsel to NLX System

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

*

Story points: 0.0
Add security e-mail address

**As a** responsible security researcher
<br>**I want to** report security issues securely
<br>**so that** the world does not abuse the issue while NLX team is fixing it

**:white\_check\_mark: Acceptance criteria**

1.  We have a "security@nlx.io" e-mail address where the world can report security issues (see #143)
1.  This e-mail address creates confidential issues in new repo `nlx-security` 
1.  This new repo has limited access rights - need to know

**:hammer\_and\_wrench: Proposed technical solution**

* Add email forwarder in TransIP

**:link: Reference links**

*

Story points: 2.0
Scheduled pipelines for security scans

**As a** DevSecOps dude
<br>**I want to** schedule a pipeline running regular scans on our containers and deployed endpoints
<br>**so that** we keep an eye on security after building / deploying

**:white\_check\_mark: Acceptance criteria**

1.  Scheduled pipeline configured in Gitlab
2.  Security scans that we use in build pipelines are added to this pipeline
3.  Results are added to Gitlab Security Dashboard

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

*

Story points: 8.0
Document Definition of Done

**As a** PO
<br>**I want** a clear Definition of Done for NLX stories
<br>**so that** it is clear to everyone when a story is truly 'done'

**:white\_check\_mark: Acceptance criteria**

1.  Definition of Done is available as an .md in repo root
2.  Issue template links to DoD
3.  Contribute.md links to DoD

**:hammer\_and\_wrench: Proposed technical solution**

*

**:link: Reference links**

*

Story points: 1.0
Document deployment flow

**As a** PO
<br>**I want** a documented deployment flow
<br>**so that** everyone involved knows exactly what to do

**:white\_check\_mark: Acceptance criteria**

1. New flow documented in contribute.md
1. Includes detailed actions for all roles, as a manual

**:hammer\_and\_wrench: Proposed technical solution**

* Add to contribute

**:link: Reference links**

*

Story points: 2.0
Adding logrecords by inway & outway is unreliable

**As a** presentor of the NLX demo
<br>**I want to** be sure the logrecords are always persisted to the database
<br>**so that** I can rely on our software

---

Currently, logrecords are not consistently written to the database. Happens to both inway (more often) and outway.

**:white\_check\_mark: Acceptance criteria**

1. Prevent the following Postgres error when inserting a logrecord to the database:

> failed to start db transaction for log insertion: read tcp 10.130.69.221:42778->10.130.60.248:5432: read: connection reset by peer 

**:hammer\_and\_wrench: Proposed technical solution**

* Reason for this to happen is unclear.
* Investigate issue, Might be the Postgres driver. Check open issues postgres drivers

**:link: Reference links**

* [Errors in Kibana](https://kibana.charlie.k8s.nlx.io/app/kibana#/discover?_g=(refreshInterval:(pause:!f,value:5000),time:(from:'2019-06-11T08:12:04.024Z',mode:absolute,to:'2019-06-11T08:27:04.024Z'))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'379f7480-85e6-11e9-897a-49a1f312e0a7',key:kubernetes.namespace_name,negate:!f,params:(query:nlx-demo-brp,type:phrase),type:phrase,value:nlx-demo-brp),query:(match:(kubernetes.namespace_name:(query:nlx-demo-brp,type:phrase))))),index:'379f7480-85e6-11e9-897a-49a1f312e0a7',interval:auto,query:(language:lucene,query:ERROR),sort:!('@timestamp',desc))).

**Screenshot**
![Screenshot_2019-06-11_at_10.33.33](/uploads/2c9f1296c9c2d160d4881dd04766d66c/Screenshot_2019-06-11_at_10.33.33.png)

Story points: 5.0
Fix Minikube after previous changes to the Helm charts

**As a** developer
<br>**I want to** run NLX on minikube again
<br>**so that** I can develop

**:white\_check\_mark: Acceptance criteria**

1.  NLX is running on minikube again

**:hammer\_and\_wrench: Proposed technical solution**

NLX is not working on minikube any more because minikube does not offer a LoadBalancer service type.

* Use Kubernetes internal DNS to redirect internal calls to internal service


Story points: 2.0
Docs: wrong description --publish parameter

In the docs https://docs.nlx.io/get-started/provide-an-api/ the description of the --publish parameter is wrong. 

`--publish 4443:4443 connects port 4443 on the host machine to port 443 inside the container. This way, we can send requests to the inway.`

should be

`--publish 4443:8443 connects port 4443 on the host machine to port 8443 inside the container. This way, we can send requests to the inway.`

Story points: 1.0