giseldo/neodataset huggingface dataset:
Meeting: Write user story on additional features for react-mosaic in joint sprint

We need to prepare some user stories that can be refined on Friday 18th during the joint-sprint refinement.

Ideas for joint sprint to go further with Mosaic:
- add empty map view as a preset (top one in the preset menu, to help users that closed all views unintentionally)
- use both light and dark themes
- other buttons to be added in the menu-bar -> Design
- customise the buttons -> Design
- show a menu when all views deleted -> Design

Other ideas:
- sync groups
- limiting the amount of screens
- limiting the width
- add map presets
- add option to auto arrange views

**Meeting: Wed 16 February 2022 at 14:00**

Story points: 3.0
FE: Prepare basic version react-mosaic

Based upon the feedback of the users, the decision was made to continue with the react-mosaic software library.

Implement a basic version of react-mosaic into geoweb.

Requirements:
* access to the current modules on Spaceweather, TAF, SIGMET, AIRMET in the KNMI GeoWeb version need to be secured (as is)
* the current presets need to be secured as well
* the add map functionality in the hamburger menu can be removed (will be replaced by the new button in a tile header bar)
* use default buttons on the menu-bar (similar as in POC)

Story points: 8.0
ADAGUC: Add extra SatComp products to geoweb for comparison

Additional products are being written to the S3 bucket but not included in the dataset. We need to add these layers so they become available in GeoWeb

* Ensure the title of the layers is descriptive so you know whether you are looking at 1km or 3km airmass for example. Also review names of layers currently already in there
* The products are already sent to the SatComp bucket and are being picked up by GeoWeb.
* We need to add the layers to the SatComp dataset.

These are the products:
* output/geoweb/png/hrv-comb/MSG_adaguc-satview-1km_hrv-comb_.png
* output/geoweb/png/airmass/MSG_adaguc-satview-1km_airmass_.png
* output/geoweb/png/airmass/MSG_adaguc-satview-3km_airmass_.png
* output/geoweb/png/day-hl-clouds/MSG_adaguc-satview-1km_day-hl-clouds_.png
* output/geoweb/png/day-hl-clouds/MSG_adaguc-satview-3km_day-hl-clouds_.png

Check if there are extra ones in SatComp which are not configured in the geoweb adaguc-server dataset.

Story points: 3.0
FE: Suppress flickering on missing images

When satellite images are missing for a time step, nothing is shown and the underlaying map is shown. When having a dedicated screen in the weather office this creates an annoying continuous flickering of the screen, because a missing image will continue to be in a looping view for at least an hour (depending on the length of the loop).

Agreed upon solution:
* reload missing images after each animation-sequence
* only solves images that could not be loaded BUT are present on the server
* really missing images will cause the animation still to flicker

Story points: 3.0
FE: Presets Cinesat Replacement

Add the following presets to the menu

Try out if you can place all 10 presets under a new drawer inside the presets menu called "Satcomp". 

**Satcomp general**

Standard domain West-Europe left window:
* HHRV Comb (not RSS)
* FIR boundaries
* Country borders
* Rivers

right window:
* Airmass RGB
* FIR boundaries
* Country borders
* Rivers

**Satcomp aviation**

left window Standard domain West-Europe:
* HRV Comb (not RSS) (domain West-Europe)
* FIR boundaries
* Country borders
* Rivers

right window:
* HRV Comb RSS (domain Netherlands)
* FIR boundaries
* Country borders
* Rivers

**Satcomp Fog ST**

left window:
* HRV Comb RSS (domain Netherlands)
* FIR boundaries
* Country borders
* Rivers (Europe)

Middle window:
* Fog
* FIR boundaries
* Country borders
* Rivers (Europe)

right window:
* HRV Comb (not RSS)
* FIR boundaries
* Country borders
* Rivers (Europe)

**Satcomp Volcanic Ash**

left window:
* Volcanic Ash
* FIR boundaries
* Country borders
* Rivers (Europe)

right window:
* HRV Comb (not RSS)
* FIR boundaries
* Country borders
* Rivers (Europe)

**Satcomp Dust**

left window:
* Dust
* FIR boundaries
* Country borders
* Rivers (Europe)

right window:
* HRV Comb (not RSS)
* FIR boundaries
* Country borders
* Rivers (Europe)

**All presets should be available in default in preset. Speed on 4. Views are synced**
- 3hr animation
- 6hr animation  (This will result in 10 presets being added - add in name "3h" or "6h")

Story points: 5.0
ADAGUC: Geographical Overlays

To make the switch from CineSat to GeoWeb, the following geographical overlays should be available in GeoWeb. These are now also available in CineSat/RAdIS/Kambeelbak. These are the ones also described in issue https://gitlab.com/opengeoweb/geoweb-assets/-/issues/1685.

Wished geographical overlays:
- Dutch FIR
- FIR delegated
- Runways EHAM AM RWY
- TMA
- NavAids
- Veiligheidsregio's NL
- AIP
- LAT/LON every 5 deg
- LAT/LON every 10 deg
- AIRPORT world
- TNCB and TNCS
- Rivers

All overlays can be added to one overlay service, the existing `Overlays`
All overlays should be in our own system, not dependent of others.


### Extra information
* all these Geographical features are available in Kambeelbak.
- RWYs EHAM: http://kambeelbak.wlm.knmi.cloud/json/geolyr/NL_EHAM_RWY.json
- Veiligheidsregio's: http://kambeelbak.wlm.knmi.cloud/json/geolyr/NL_veiligheidsregios.json
- NAVaids currently available in RADIS

Story points: 5.0
ADAGUC: Combine all overlays in one overlay datalayer

### User

Meteorologist

### Short description

Currently, we're using to different services for overlays `Overlays` and `SigmetAirmetOverlays`. Pending final decisions on how to get overlays in the map, these services need to be combined.

In both services, the layer `Stations North Sea` and a the layer `Airports` exist. The different stylings in these services must be merged in one layer with multiple styling, so one `Airports` layer with two stylings; Icon and Circle. The default for each layer must be the circle (actually a circle with a centerpoint).

### Test criteria

* Open overlay service
* user can select different styles for overlay-layers (e.g. either circle or icon)
* default styling is circle

Do:
- In geoweb-adaguc, remove the sigmetairmetOverlays dataset and move the layers inside to the Overlays dataset. Ensure that the duplicate names get merged together and all styles currently available are still available but now under one and the same layer and service.

Story points: 3.0
Change serviceId from URL to id in redux

Refactor of services store part of the user story.

Changes included:

* service object in services store should have
  * unique id as its identifier instead of URL
  * id property deleted
  * service property could be renamed to url or serviceUrl
* Refactor all code that uses the "id" of services array for anything related to it being an URL to use the "service" variable inside the service object instead (atleast getting info from getCapabilities)



Next steps are:
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1883
- https://gitlab.com/opengeoweb/geoweb-assets/-/issues/1864

Story points: 8.0
Groups are saved to services store instead of Layers inside them

Currently in LayerSelect, if service has its layers separated to different groups, only the group is added as a layer to the redux store and shown, instead of the actual layers that are inside that group.

* In getCapabilities groups are Layer objects, probably why this is happening. 
* In the old WMSLoader, when adding service to a store through adding a layer to the map, it does work properly and serviceSetLayers action is able to add the real layers to the store.
* When layers in groups are added correctly, the groups are actually stored inside the layers "path" variable, but very badly, groups and subgroups are just put together to one string, without any sign of how many subgroups there are, in this example, there are 3 groups/subgroups for the layer: Fachlayer, Wetter and Warnungen 

![image](/uploads/cffd5e7af18420e89535d3b7bf7fb58b/image.png)

To fix the issue, we need to do couple things:
* Investigate why serviceSetLayers works differently in LayerSelect compared to WMSLoader and fix it in LayerSelect
* Refactor layers path variable to have better format, from which different groups can be identified clearly, I suggest renaming it to "groups" and having it as a array of strings
* Either Block groups from appearing in services layers array, or handle them differently by using the leaf variable, which is false for groups and true for layers

Story points: 5.0
FE: fix npm vulnerabilities 2: update typedoc, license, msw, postcss, browserslist

These packages are causing vulnerabilities and have new versions available. Install latest versions for:

- [x] typedoc
- [x] license-check-and-add
- [x] msw
- [x] postcss
- [x] npx browserslist@latest --update-db
- [x] axios
- [x] check for breaking changes

Do NOT update:

1. cypress + @types/testing-library__cypress --> wait with upgrading until issue resolved https://github.com/cypress-io/cypress/issues/18464
1. @babel/cli + @babel/core --> breaks all snapshot jobs and ServiceList test, see https://gitlab.com/opengeoweb/opengeoweb/-/jobs/2067700947

Story points: 3.0
FE: Fix npm vulnerabilities 1: update @storybook + @nrwl

Investigate which packages can be updated and if there are dependencies on other packages that need updates first.
Split into multiple tickets if necessary


Packages that need updating:
1. @storybook + @nrwl + @svgr/webpack (dependent so need to update together)

To be updated in other [ticket](https://gitlab.com/opengeoweb/opengeoweb/-/issues/1770):
1. @babel/cli + @babel/core
1. typedoc
1. license-check-and-add
1. msw
1. postcss

No update:
1. cypress + @types/testing-library__cypress --> wait with upgrading until issue resolved https://github.com/cypress-io/cypress/issues/18464

Story points: 3.0
FE: investigate Improve selector memoization to reduce unneccesary renders

Suggestion from Maarten:
- Create a story with two maps
- Change the opacity of one of the layers on one map, the other map should NOT rerender

Investigation for the following:
-----------------------------------------------------------------
**Improve selector memoization to reduce unneccesary renders**

**Description**: Consider mapA and mapB each having a different set of layers. The issue is that changes for a layer in mapA will trigger unneeded renders of mapB. For example when you set opacity for a layer in mapA, mapB (and all other maps) is also re-rendered.

**Problem**: It seems that the getMapLayers selector causes unneeded re-rendering of components. This probably is because the layers are in one redux state object. It is probably difficult for the selector to detect which changes affect what maps.

**Solution**: Make sure that when the result of the selector is unchanged, the component does not re-render. This has to do with memoization and shallow comparison of objects.

**Hint**: From file libs/core/src/lib/store/mapStore/map/selectors.ts, 122: the selector getMapLayers is used in libs/core/src/lib/components/MapView/MapViewConnect.tsx, 86 . It seems that changes in the layers for mapA, will trigger unneeded re-renders for mapB. This becomes an issue when there are many maps in view, like with the multimap component with 18 maps.

Story points: 3.0
FE: Performance: Improve selector memoization to reduce unneccesary renders.

**Improve selector memoization to reduce unneccesary renders**

**Description**: Consider mapA and mapB each having a different set of layers. The issue is that changes for a layer in mapA will trigger unneeded renders of mapB. For example when you set opacity for a layer in mapA, mapB (and all other maps) is also re-rendered.

**Problem**: It seems that the getMapLayers selector causes unneeded re-rendering of components. This probably is because the layers are in one redux state object. It is probably difficult for the selector to detect which changes affect what maps. 

**Solution**: Make sure that when the result of the selector is unchanged, the component does not re-render. This has to do with memoization and shallow comparison of objects.

**Hint**: From file libs/core/src/lib/store/mapStore/map/selectors.ts, 122: the selector getMapLayers is used in libs/core/src/lib/components/MapView/MapViewConnect.tsx, 86 . It seems that changes in the layers for mapA, will trigger unneeded re-renders for mapB. This becomes an issue when there are many maps in view, like with the multimap component with 18 maps.

Story points: 3.0
FE: Performance: layerSetStyles info should belong to service metadata

**layerSetStyles info should belong to service metadata**

In function setServiceMetadata, in ReactMapView, 355, layerSetStyles are currently set for each individual layer instance. It would make more sense to have a service metadatastore with layer information, storing all possible styles and dimensions for the layer. The layermanager can use these for filling the dropdowns. 

Proposal is to move this layer information to the service store. This needs to be set only once and during refresh. The information can be re-used for same layers in other maps.

Please discuss with FMI on the impact of this change to their new components.

Story points: 5.0
FE: Performance: Combine layer setServiceMetadata actions

**Combine multiple actions to set metadata for layers into one**

The function setServiceMetadata in ReactMapView, 355 should be rewritten to trigger one action updating the redux state at once. This will result in less re-rendering of components. Currently this triggers 5 actions for each layer. This causes 5 re-renders of the map component (each containing multiple layers). In https://gitlab.com/opengeoweb/opengeoweb/-/issues/1670 it was found that with multiple maps, over 10000 renders take place before the final display.

Combining multiple actions into one will result in significantly less re-renders.

Story points: 5.0
MUI5: fix date picker functional + unit test

- Fix unit test: instead of click on OK, use the pickers to choose a date and time like the user would do. (unit test is commented out)
- Make sure it behaves exactly like currently spaceweather, if you hover hour / minutes it should update the TextField. --> not able to fix this, it also happens on the mui docs: https://mui.com/components/date-time-picker/. I've only seen it work with the static time picker https://mui.com/components/time-picker/#static-mode

Styling should be fixed by the form-fields, as the DatePicker is using a TextField

Story points: 5.0
MUI5: review current state (functional)

Review current state of branch https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1015/diffs

- if the imports are working now (no tree shake), then we need to update the Readme as well
- Slider en DatePicker have commented out (test) code, that can be ignored 
- check out the project.json files with external files (no old material-ui)

Story points: 2.0
PO Roadmap review

In the Intl. review and KNMI retro (14 Jan 2022) PO(s) explicitly ask developers to review the RoadMap in which they are working. During the retro we took this as task to be done in Sprint 59.
You can write a comment if you think something is missed or you want to ask something.

The roadmap is here:
https://gitlab.com/opengeoweb/geoweb-assets/-/issues/1695

Every one of us should at least look at it because this is what PO(s) are thinking to do in the coming year(s).
I will assigne this task to everyone. Once you reviewed the document, please remove also your name from the assignees list so we know who is still missed. Thanks!

Story points: 1.0
FE: clouds not validated correctly if space present

If you enter a space in one of the cloud fields, the other cloud fields are not validated correctly anymore:

For example - below it has "NSC " (so with a space) in the first cloud group. Now it evaluates the third cloudgroup as ok while it is actually wrong. 
![image](/uploads/0f96ffccf24d2a0dfbfaa5af20927448/image.png)

Also the order of validation is wrong. In the below example, it is more important to say that the field should be empty after NSC than complain about the format being wrong cause after you fix the format you'll then still be notified it is not allowed after NSC
![image](/uploads/31b53f30d2703ac168aad19454da4209/image.png)

Story points: 3.0
Clean up backlog

Remove tickets no longer needed as they are for (soon to be) deprecated components (Check list of “Tickets already done”)

- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1488
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1335
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1198
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1588
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/1589
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/623
- https://gitlab.com/opengeoweb/opengeoweb/-/issues/957

Story points: 3.0
FE: TAF: Cloudcoverage in groups when CB or TCU present.

:drum: A group with CB or TCU is allowed at the end or between changegroups. This group with CB or TCU can have any cloud coverage type (ov, sct, bkn, few).

Details in https://gitlab.com/opengeoweb/geoweb-assets/-/issues/1635

Story points: 5.0
FE: TAF changegroup not removed when switching forms

There is a bug where the changegroups for tafs with empty changegroups are not correctly cleared out when switching TAFs from one with filled in changegroups.

You can see in the movie that the EHLE TAF has no data in the changegroup. When moving to the EHAM TAF which does have data and then back to the EHLE TAF the EHAM changegroup data remains visible instead of being cleared out. [movie](/uploads/c333231c3977cde9ffc38acb178a5e5d/TAF2.mkv)

NOTE: This only is the case for TAFs that have NO empty changegroup (so drafts and completely new TAFs are not affected)

Story points: 3.0
BE: SpaceWeather streams table init incorrect

The script cfn-init-streams.py incorrectly initializes the streams table: it should store 1970-01-01 in the end time and 2050-01-01 in the start-time, but does it the other way around.
The front-end uses the end-time field for automatic updating of the timeseries, which never happens because the end-time never gets updated when new data comes in (reqSW/reqX, etc).

To avoid future mistakes like this I've renamed the constants for MAX_TIME and MIN_TIME.

Story points: 2.0
FE: sigmet loader is showing when closing sigmet dialog

If you open a sigmet dialog, and go back it's refetching the list call and showing the loader. The list call should load initially, but not after opening a sigmet and going back to the list overview without taking any action such as save/discard/publish/cancel

![Screenshot_2023-01-10_at_10.58.39](/uploads/bdce69e95be2e1c60479015c75d56832/Screenshot_2023-01-10_at_10.58.39.png)

Story points: 2.0
FE: valdation for sigmet/airmet config file

We need to add validation for the two config files.

- If they are invalid - show a decent error on screen. Ensure the system doesn't crash and tell there is an invalid config (maybe also display what is wrong/missing/invalid).
- Also add unit tests to test the keys in the config file and check that the error above is displayed correctly
- Have a look and see if we can reuse (or use as inspiration) how this is dealt with in the app

Should be independent of institute and be placed inside of the sigmet/airmet lib

Story points: 3.0
FE: sigmet/airmet - Use FIRs from config to draw on map

On opening of the form all of the FIRs geojsons should be shown. Go through the config and extract all FIR geojsons to show. Add them to one new maplayer in the map.

- Add this new maplayer in MapViewGeoJson.tsx as `MapViewLayer` or add each FIR as a separate `MapViewLayer` (see which is easiest)
- Ensure that the functionality regarding the currently selected FIR remains UNCHANGED

Story points: 3.0
FE: Edit form of Service popup not working anymore

The LayerSelect -  SerivcePOpup edit mode doesn't work anymore. The save button is permanently disabled.

I did some digging and the fields in the react-hook-form remain undefined even though there is data set using the reset().

Typing also doesn't trigger any rerenders or changes within the form-fields

For more explanation see: https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/2108#note_1218499201

Story points: 3.0
FE: fix ReactMapView error on Sigmet when map has multiple layers

- go to homepage
- add a minimum of 2 layers to the map or open a preset that has at least 2 layers on the map
- now open the Sigmet module
- click Create a new SIGMET
- this gives `RangeError: Maximum call stack size exceeded`
- if you have 1 or zero layers on the map, all works fine
- it also works fine on tst, so something broke since the last release

![image](/uploads/77d3a1f71b9ad98a048943ea5dddbb60/image.png)

Story points: 2.0
FE: investigate if we can clear store on navigating stories

The presets stories don't work correctly after navigating, as presets are fetched initially and stored globablly. A story works if if you refresh, so nothing is broken.

Find out if we can clear the store on navigating stories. To test, try out navigating between these two stories, should work correct (presets):
- http://localhost:4300/?path=/story/application-demo--presets-demo-with-error-on-fetch-list
- http://localhost:4300/?path=/story/application-demo--presets-demo-on-map-dark-theme

the solution should be used for presets, but maybe handy for other libs as well?

Story points: 2.0
FE: React 18 sigmet level values that are saved are not displayed in the form

When opening a sigmet from the list that has a valid levels value saved, in React 18 strict mode the levels value shows empty and is autofocused. Instead it should show the saved value.

See if the same fix works for https://gitlab.com/opengeoweb/opengeoweb/-/issues/3102

Story points: 3.0
FE: React 18 spaceweather/notification/populate second request fails with invalid request body

When opening a notification from UKMO that can generate a KNMI notification (for example a Geomagnetic impulse alert or summary), the application does a spaceweather/notification/populate request to autofill the KNMI notification fields. In React 18 with strict mode this request is done twice, and one fails because it's missing a lot of the parameters (BE will send error `Invalid request body`). Some fields are displayed empty while they should be filled. Seems like at first render they are filled, but after unmount and mount they are not anymore.

Maybe same solution will work as used for https://gitlab.com/opengeoweb/opengeoweb/-/issues/3103?

Story points: 3.0
FE: drag and drop in layermanager and taf change groups is broken in React 18

we use react-sortablejs for drag and dropping layer rows in layermanager and changegroups in taf form, this doesn't work anymore in React 18. There is no new version yet, there was an issue logged 2 weeks ago but no replies yet.

https://github.com/SortableJS/react-sortablejs/issues/263

This might be the MR with the fix: https://github.com/SortableJS/react-sortablejs/pull/265/files

Works fine if strict mode is disabled.

Story points: 5.0
FE: fix React 18 login issue - token call cannot be done twice

In Code.tsx (authentication lib) there is a UseEffect with empty dependency array that does the token call on first render. With React 18 strict mode this will be fired twice. The second token call fails and breaks the login. This call should only be done once. Can be solved in different ways, see other api code (use a ref, use an abortcontroller, etc.). 

Preferred solution is using abortcontroller. First request should get cancelled.

The React 18 MR already contains a fix to be able to test other things, but there might be a better way. See https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/2034/diffs#ca1564fb2e5360b04153a34b7703ff4445f55b75

Story points: 3.0