giseldo/neodataset huggingface dataset:
Clean-up Sesar

Sesar is no longer used, so we can clean it up.

To do:
- [x] Archive repositories (I cannot find any old repositories anymore)
- [x] Remove Logstash filters
- [x] Drop CFN stacks
  - [x] Userpool dev: https://eu-west-1.console.aws.amazon.com/cognito/v2/idp/user-pools/eu-west-1_kqPqg3g7p/users?region=eu-west-1
  - [x] Userpool acc: https://eu-west-1.console.aws.amazon.com/cognito/v2/idp/user-pools/eu-west-1_94RP6yZBi/users?region=eu-west-1
  - [x] Sesar dev s3 bucket: https://s3.console.aws.amazon.com/s3/buckets/knmi-sesar-data-sp10-test-memory?region=eu-west-1
  - PRD:
  - [x] Sesar cloudformation shh gateway: https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/stackinfo?filteringStatus=active&filteringText=sesar&viewNested=true&hideStacks=false&stackId=arn%3Aaws%3Acloudformation%3Aeu-west-1%3A076350057214%3Astack%2Fknmi-sesar-SshGateway-IJI8OBWJ7DHY%2Fce087290-2d5d-11ea-9723-0aac2484be2a
  - [x] Sesar cloudformation backend 100day archive: https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/stackinfo?filteringStatus=active&filteringText=sesar&viewNested=true&hideStacks=false&stackId=arn%3Aaws%3Acloudformation%3Aeu-west-1%3A076350057214%3Astack%2Fknmi-sesar-SesarBackend100DayArchive-4HPBWGU9ULWD%2Fce112520-2d5d-11ea-8bc7-02bbe9b74d60
  - [x] Sesar cloudformation product 100day archive: https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/stackinfo?filteringStatus=active&filteringText=sesar&viewNested=true&hideStacks=false&stackId=arn%3Aaws%3Acloudformation%3Aeu-west-1%3A076350057214%3Astack%2Fknmi-sesar-SesarProduct100DayArchive-5BPWKABOTIBA%2Fcdb79370-2d5d-11ea-af61-063a4310b160
  - [x] Sesar cloudformation app:https://eu-west-1.console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/stackinfo?filteringStatus=active&filteringText=sesar&viewNested=true&hideStacks=false&stackId=arn%3Aaws%3Acloudformation%3Aeu-west-1%3A076350057214%3Astack%2Fknmi-sesar%2Faf4dc760-2d5d-11ea-990f-0ae0d664c880

Story points: 3.0
Layer listing is displaying last selected layer instead of service name

### Steps to reproduce

* open new layer select
* select to display only one service
* add one layer to map using Add button
* see how service name is now added layer name in all the rows


### What is the expected *correct* behavior?

Service name in the listing should not be affected

### NEXT STEPS
Check if this bug is still happening after this fix: https://gitlab.com/opengeoweb/geoweb-assets/-/issues/2223
If bug is still happening create a new ticket to solve that bug

Story points: 2.0
BE: security check less strict in all pipelines

Make the security check less strict in all pipelines (DEV, TST, ACC and PRD). It means that a vulnerability presence (Critical or high) will not block the pipeline but a message will be sent per mail or slack (tbd) with the vulnerability warning description.

- Less strict security check
- Send message per mail/slack in case of vulnerability warning


Documentation for build-image / ecr.sh: https://gitlab.com/KNMI/shs/knmi-build-image

Story points: 3.0
BE: CPT discussion on Vulnerability vs Deployment strategical implementation

Have a discussion with CPT on the vulnerability handling

To discuss:
1. ask CPT to decrease the vulnerability report to 7 days (instead of 21 as currently)
2. ask if possible to do a scan every day (or some other frequency that is appropriate) instead of on push only
3. Discuss our plans regarding vulnerability handling following https://gitlab.com/opengeoweb/geoweb-assets/-/issues/2036


------
**Conclusions so far:**
1. It is **not possible the decrease the vulnerability report to 7 days**
- For the vulnerability report, the last image in each ECR repository is scanned
  - If in this image a vulnerability is detected, it is assumed that this vulnerability will be present in the older images in the ECR repository as well.
  - This assumption is also valid for active containers
 
2. In between vulnerability reports (every 21 days), the last uploaded ECR images are **scanned once a day** (but a lambda collects the information every hour, see below). The results of these scans are visualized in Grafana
  - DTA dashboard: https://grafana.dta.logs.knmi.cloud/
  - PRD dashboard: https://grafana.prd.logs.knmi.cloud/
  - Attached to these dashboards are alerts set for detecting critical vulnerabilities. This provides alerts in between 21 days of the report

3. After a discussion of our plans, it is discouraged to make security less strict in all pipelines. CPT prefers to have high and/or critical vulnerabilities failing the pipeline (or the thresholds)- this sends a stronger message than letting the pipeline pass & sending an alert - as it motivates developers to immediately fix the vulnerability.
  - Regarding vulnerabilities emerging during deployment to production: if vulnerabilities are found in images in development/acc, one can assume that this vulnerability will occur in all images and should be patched before going to PRD as well. 
  - Regarding sending out an alert after a vulnerability has been found during the pipeline: an alert should be sent out within 24 hours when detected via the Grafana dashboard. Therefore, setting the pipeline less strict could be an option, but it might be some additional work to send out an alert via the pipeline if it will also be detected by the Grafana dasbhoard with a short delay.


------
Follow up: make a ticket to discuss how to monitor vulnerabilities in PRD / active containers in the cluster

Story points: 3.0
Layer Select info layer part 2

Create the content of the dialog. Thumbnail, legend and share link can be ignored. Make sure the layer icons are in as well. 

If there is not enough data, just create a story with fake data and focus first on styling. Create a new ticket for hooking up with real data. 


Light (https://zpl.io/JEzogXJ + https://zpl.io/5E6PoMQ) Dark (https://zpl.io/qyAZdJJ + https://zpl.io/GE6k8JE)

Story points: 3.0
Layer Select info layer part 1

- Add the i Icon to the list 
- implement ToolContainer Dialog 
- content should be a string of "Hello World"
- Only one info window open at one time. If you click next info button, content updates. If you close Layer Select window also info window closes.

Light (https://zpl.io/JEzogXJ + https://zpl.io/5E6PoMQ) Dark (https://zpl.io/qyAZdJJ + https://zpl.io/GE6k8JE)

Story points: 2.0
BE: Use TST userpool for master and dev stacks

At this moment there is a DEV userpool for dev stacks and a TST userpool for master stack. The DEV userpool can be removed and only the TST userpool can be used for both dev and master stacks

- Rename current DEV userpool in https://gitlab.com/KNMI/geoweb/backend-services/dev-cognito-userpool/-/blob/master/cloudformation.cfn.yaml and remove the TST userpool from serverless BE. 
- Rename the above repository to "cognito-userpool"
- TST userpool is updated on a merge to master

### Test criteria

* DEV userpool is removed
* TST userpool is used for both dev and master stacks
* ACC and PRD are left as is and won't be affected

Story points: 3.0
Layer Select: Enable and save Abstract part 2

When saving or editing a service, we should update the information to redux and localstorage also.
If browser is refreshed, the services are fetched from localStorage and added back to redux

ToDo

- Save abstract into redux
- edit saga and save abstract into localstorage also
- If browser is refreshed, the services are fetched from localStorage and added back to redux

Story points: 3.0
FE: see if we can enable embedding via https in storybook

If you check the console, the following error appears:

![Screenshot_2022-09-19_at_13.15.32](/uploads/128521f259eb0e3c4e9a2bb4d94e80f9/Screenshot_2022-09-19_at_13.15.32.png)

Probably the problem is caused by the zeplin link is one with `https` and the story preview is not. See https://support.zeplin.io/en/articles/5671951-troubleshooting-tips-for-storybook-integration for more information. Probably we need a `ssl certificate` to run storybook with https. (try `nx storybook theme --https`). More info in nx: https://nx.dev/packages/angular/executors/webpack-server#ssl 

If this solves the problem, see if we can use the zeplin URI links. Might be that this is related, and if it works create a new ticket to update the zeplin links (not needed per se but might as well do it since the total is not that big) and readme.

Story points: 2.0
FE: storybook set default orientation to right

Currently when starting a storybook, the addon panel is always default orientated to the bottom. 
- We can change that to right for default with `panelPosition`. 
- Also default the panel to hidden

Create a new manager.js file in storybook and change this to right https://storybook.js.org/docs/react/configure/features-and-behavior

Story points: 2.0
FE: Replace missing satellite layers in presets

Layers in preset are not available anymore, since cinesat is stopped. Preset can be removed.

After stopping cinesat the following presets have some layers that are not working anymore:

* Satellite
* Radar and Satellite
* SIGMET/AIRMET
The none working satellite layers in these presets can be replaced:

**Satellite** (add updated single map preset only to map preset menu in layermanager and remove from hamburger menu) missing `HRV RGB` replace by Satcom HRV comb RGB

**Radar and Satellite**

??? (currently saying `no service available`) replace by Satcomp RSS HRV comb RGB

**SIGMET/AIRMET** (add updated single map preset only to map preset menu in layermanager and remove from hamburger menu)

missing `HRV RGB` replace by Satcomp HRV comb RGB

missing `vulkaanas` replace by Satcomp Ash RGB 



Actions in FE:
- update **Radar and Satellite**
- Remove **Satellite** and **SIGMET/AIRMET** from `screenpreset.json` in `knmi-geoweb` and `knmi.screenpreset.json` in `geoweb`

Story points: 2.0
FE: fix rest of MapPreset props part 3 animation payload, settimestep, toggleTimeStepAuto

These give a lot of issues. Mainly because duration coming from animation payload can also be set by setTimeStep. First thing should be to get the preset a bit cleaner. Afterwards make it work with saving. 
```
 animationPayload?: AnimationPayloadType;
  setTimestep?: number;
  toggleTimestepAuto?: boolean;
```

Check if these still need to be defined inside MapPreset type:
```
  displayMapPin?: boolean;  
```

- shouldShowZoomControls https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1681


See https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1646 for reference

Make sure to do this after [part 2](https://gitlab.com/opengeoweb/opengeoweb/-/issues/2335), because some props are very intertwined with each other

Story points: 5.0
FE: dimensions not loaded correctly from preset

Create a preset with an elevation dimension for example and set the pressure level to 700.

- Load the preset. 
- You will see it selects the minimum elevation (200) instead of the 700 from the preset. The values are stored correctly in the BE but the FE overrides as it hasn't yet loaded the dimension values upon loading the layer.
- in layer update information should update the current (stored) dimensions

Story points: 5.0
FE: app crashes when selecting some presets with out of sync dimensions

To test:

* `nx serve knmi-geoweb`
* from service Harmonie N25, add layer `Geopotential height (PL)`
* save as custom preset
* now reload the app by refreshing your browser
* open the preset immediately
* browser is crashing



![Screenshot_2022-07-25_at_12.57.08](/uploads/c4ccd02bb832c0881b475e4010683ddd/Screenshot_2022-07-25_at_12.57.08.png)


If we don't store the dimensions with name time or reference_time, it won't give any issues. Filter those dimensions out in the selector `getMapPreset`. This tryout worked:
```
export const getMapPreset = createSelector(
  getMapLayers,
  getMapBaseLayers,
  getMapOverLayers,
  getBbox,
  getSrs,
  getActiveLayerId,
  (
    layers: Layer[],
    baseLayers: Layer[],
    overLayers: Layer[],
    bbox: Bbox,
    srs: string,
    activeLayerId: string,
  ): MapPreset => {
    const allLayers = [...baseLayers, ...overLayers, ...layers].map((layer) => {
      const dimensions = layer.dimensions.filter(
        (dimension) =>
          dimension.name !== 'time' && dimension.name !== 'reference_time',
      );

      return {
        ...layer,
        dimensions,
      };
    });
    return {
      layers: allLayers,     
    };
  },
  selectorMemoizationOptions,
);
```

Story points: 3.0
BE: fix CORS issue when reading header value "Location"

In the frontend, we want to read the ID of a newly created mapPreset. The data is already there, we can use the 
`Location` key from the response headers. Only when trying to access it, this error appears:

```
Refused to get unsafe header "Location"
```

A suggestion to fix it can be found here https://stackoverflow.com/questions/7700151/refused-to-get-unsafe-header-location. If you want to try it out in the front-end, see the code described in https://gitlab.com/opengeoweb/opengeoweb/-/issues/2329

Story points: 2.0
FE: fix save preset as flow

If you press `save As` in the preset option menu, the flow should be (fix mentioned in bold):
- **close the menu with preset list**
- open the dialog (already present)
- choose a title and press save (already present)
- **now get the newly created Id from the header response, and activate this preset as activePreset**

We can get the new id from the header response `Location` of the saveAs post request:

![Screenshot_2022-07-22_at_12.56.23](/uploads/579ee65bb894532b8633b016f27337a6/Screenshot_2022-07-22_at_12.56.23.png)


Inside `utils/api`, do to try out if we can read the value:
```
 saveViewPresetAs: (data: ViewPreset): Promise<void> => {
      return axiosInstance.post(`/viewpreset/`, data).then((response) => {
        return response.request.getResponseHeader('Location');
      });
    },
```

After this we can hook it up inside `MapPresetsDialogConnect` to handle the correct new active id

Story points: 3.0
FE: add the WOW WMS service to the knmi list of available services in knmi-geoweb and geoweb


add the WOW WMS service to the list of available services in knmi-geoweb and in the knmi.initialPresets in geoweb:
(in both places because we are not yet using unified geoweb for deployments)

https://adaguc-server-geoweb.geoweb.knmi.cloud/adagucserver?dataset=WOWDATA

See https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1735

Story points: 2.0
Make space weather doc for Met Office

Met Office: The spaceweather API for notifications is being updated and Met Office would like to know our wishes and which fields we use as they can be a bit flexible. We agreed to create a short document with our needs/wishes so they can take them into consideration. More info here: https://docs.google.com/document/d/1853HaJTTzABt8F1qk8CNpmvQmycBIGHIvS4gLVtfs5o/edit

Story points: 3.0
FE: fix rest of MapPreset props

In https://gitlab.com/opengeoweb/opengeoweb/-/issues/2259 some basic props are working when changing the mapPreset (bbox, layers, overlayers). fix the rest of the props, check the MapPreset props to see, and add the missing selectors to the getMapPreset selector

- see changes
- make possible to save
- bundel props that belong to eachother 


IN:
```
  shouldAnimate?: boolean;
  shouldAutoUpdate?: boolean;
```

Rest of props (https://gitlab.com/opengeoweb/opengeoweb/-/issues/2335):
```
  shouldShowZoomControls?: boolean;
  showTimeSlider?: boolean;
  displayMapPin?: boolean;  
```

Problem props, all related:
```
  animationPayload?: AnimationPayloadType;
  setTimestep?: number;
  toggleTimestepAuto?: boolean;
```

Story points: 3.0
BE: Investigate why in the presets backend request.get_json returns None

Via nginx and gunicorn, the function `request.get_json` is returning `None`. We had to use `json.loads(request.data.decode('utf8'))` as a fallback.

We suspect some required headers are not correctly forwarded from nginx to gunicorn. Worthwhile investigating as this get_json function is indeed better.

Story points: 3.0
BE: Preset BE should return informative message on error

When the Preset BE "Save As" and "Save" functionality fails with a 400 code, it is unclear for users what the reason is. It can be because of a malformed JSON, it can be that the preset name already exists. 

Users should be presented with more informative messages in the case of errors.

Example use case:
- "Save As" with duplicate title

Story points: 3.0
FE: more icons for sigmet-airmet

Unfortunately some icons were missed in the https://gitlab.com/opengeoweb/opengeoweb/-/issues/2279.

```
import {
  Create as CreateIcon,
  CheckCircle as CheckCircleIcon,
  Schedule as ScheduleIcon,
  InfoOutlined as InfoOutlinedIcon,
} from '@mui/icons-material';
```
should also be in the Icons lib and imported from there.
Check also if there are other icons in sigmet-airmet library

Story points: 2.0
FE: cleanup stories in presets

- The default story was `http://localhost:4300/?path=/story/components-mappresetsdialog--map-presets-dialog-dark-theme`, not sure it was the best one, but would be nice to put that back, as it gives a nice overview of all currently implemented options.  Otherwise `http://localhost:4300/?path=/story/components-mappresetsdialog-mappresetsdialogconnect--map-presets-dialog-on-map-dark-theme` is also nice on the map, but then with default layerManager open so it saves a click
- The stories are hard to read in a list, better to categorize some stories to for example state stories. 

![Screenshot_2022-07-13_at_12.10.55](/uploads/79d43c7e19f1b54a4247fe44cab85c81/Screenshot_2022-07-13_at_12.10.55.png)

Story points: 1.0
BE: change value/use of preset_type field in presets database

The preset_type field in the view_presets table currently has values of global or user.
This is not used anywhere and seems to serve no purpose.
This information is already covered by the scope field ("system" or "user")

The preset_type field should actually contain a value "map" for map view presets.
This should be the default value.
This should also be changed in the values already in the database.

Story points: 2.0
FE: show error for preset detail

Show the error in layermanager when fetching of a preset goes bad. For now show a error, with a close icon to remove it. Later we might remove the error state to redux after we can see changes in an active preset. 

https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/61eff734e45296a2b037345f

![image](/uploads/e6e54c900db00faca65f40453f60a9cd/image.png)

Story points: 3.0
FE: loading state load preset detail

Show loading state when clicking on a preset. Use progress bar inside the layermanager. https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/5ecf926f2711dd29d9dca7b3

https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/61eff734e45296a2b037345f

Story points: 2.0
FE: loading state load presets list

Show loader when fetching the preset list. Make two versions (POC) and show to team/Didier and pick one. After that create the unit tests.

--> Didier choose the spinner and design can be found here: https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/61eff734e45296a2b037345f

Story points: 3.0
BE: Save a user preset

- Save a user preset. 
- This will overwrite an existing user preset in the backend. 
- API definition is discussed in https://docs.google.com/presentation/d/19EZolb7x2DDhQg3tkEs8ejlKaKCq7Qni6XQ7XWKAJ9Q/edit#slide=id.p

Story points: 2.0
FE: Rewrite service store reducer with redux-toolkit

Acceptance criteria:
- Replace reducer in libs/core/src/lib/store/mapStore/service/reducer.ts with createSlice reducer from react-toolkit
- Use actions from createSlice reducer in all affected code. Remove actions.ts and constants.ts file.
- Use reselect and redux from redux-toolkit and delete dependencies from package.json

Story points: 3.0
BE: Delete a user preset

It should be possible to delete a user preset. Implement this according to the discussed API. 

- Discuss how the API can be called to delete a preset (DELETE rest command)
- Implement in presets backend

Discussion with BE devs. Write down the spec in the document. (https://docs.google.com/presentation/d/19EZolb7x2DDhQg3tkEs8ejlKaKCq7Qni6XQ7XWKAJ9Q/edit#slide=id.p)

Update the Readme.md as well

Story points: 3.0
FE: Rewrite layer store reducer with redux-toolkit

Acceptance criteria:
- Replace reducer in libs/core/src/lib/store/mapStore/layers/reducer.ts with createSlice reducer from react-toolkit
- Use actions from createSlice reducer in all affected code. Remove actions.ts and constants.ts file.
- ~~- Use reselect and redux from redux-toolkit and delete dependencies from package.json~~ Moved to https://gitlab.com/opengeoweb/opengeoweb/-/issues/2271 because it will be a smaller MR

Story points: 5.0
Delete unused ResizeHandle component from screenmanager

The `ResizeHandle` component has been replaced by `DraggableResizable/Resizable` in `shared`, so the unused component in `screenmanager` can be deleted.

Delete this component:
https://gitlab.com/opengeoweb/opengeoweb/-/tree/master/libs/screenmanager/src/lib/components/ResizeHandle

Story points: 1.0
FE: add keyboard control for confirmation dialog

The confirmation dialog currently has no keyboard support. Would be nice to have the following support:
- esc -> close dialog: add eventlistener and remove on unmount
- enter -> submit dialog: autofocus on submit button when opening the dialog

If the focus is on the cancel button and you press enter, it should still cancel. Tab and shift-tab should still work. The focus should be visible.

Story points: 3.0
Fix GOES irregular product time issues

The NOOA GOES-16 data has a slightly varying observation time: the times can deviate one or more seconds from a regularly repeating sequence.
This has to be fixed in ADAGUC (by means of implementing quantizemethod and quantizeperiod also for observation times).
Then the fixed ADAGUC version has to be used in GeoWeb-Adaguc-Server (by updating the used nested stack version) and configured in the SAT.xml configuration of GOES-16 data

Story points: 3.0
FE: delete preset

Delete preset in fe, connect:

* api
* fakeApi
* remove disabled prop in MapPresetsMenuItem and uncomment the unit tests
* show confirmation dialog by using current implementation (we could not use the useConfirmationDialog hook, as the form needs some extra fields, and a promise only returns a value once). use `action` prop to remove the `content` prop for this action, and toggle onSubmit method to not validate form 
* enable unit tests again

Story points: 3.0
FE: Generate service_id in WMJSService instead of reducer

# FE: Generate service_id in WMJSService instead of reducer

### Note:  This is for the layerselect component.

The layerselect store needs the same id's as the service store. Therefore the id needs to be generated in  WMSJSService. This will make the same id available for all stores. It also allows for finding the id of a service by looking it up via its url using the `WMGetServiceFromStore` function from `webmap`.

Story points: 1.0
BE: Presets backend: Configure new database SQLAlchemy url

When using sqlalchemy, a different database connection string is needed.

We need to change it at https://gitlab.com/KNMI/geoweb/backend-services/deploy-presets-backend/-/blob/master/cloudformation.cfn.yaml#L190

For docker-compose the change was from :
```
      - "PRESETS_BACKEND_DB=host=presets-backend-db port=5432 user=geoweb password=geoweb dbname=presetsbackenddb"
```
to
```
      - "PRESETS_BACKEND_DB=postgresql://geoweb:geoweb@presets-backend-db:5432/presetsbackenddb"
```

Something similar needs to be done in cloudformation.

Story points: 1.0