giseldo/neodataset huggingface dataset:
SpaceWeather: Make a plan on how to keep the KNMI notifications during updates of the backend database

Have a meeting and make a plan:
- After re-ingesting Space Weather UKMO data, KNMI notifications are lost. 
- Should we save these separately for re-ingesting (coupled by UKMO notification or uncoupled "KNMI-only")?
- There is still a notifications database lingering around.

UKMO Notification comes in and is stored in the Incoming notification table. At KNMI this Notification is stored in the Event table and presented as an Event in the application. KNMI can add information and forward the KNMI Notification (stored in Event table) to DCC. KNMI can also make events on its own.

Note: 
- The incoming notification table is never affected by changes from KNMI, but the Event table is affected.
- The Event table contains the notification and extracted information from the notification

**Problem**: The event table should become persistent. The format of the events changes during development. How to we get the old events in the newer format? Currently we just throw away the database to prevent errors happening in the backend or frontend due to missing fields.

**Solution A**: Migrate the Event table database by using a backup/restore operation. Make a backup of the Event table and clear it. Run a restore script to add the existing events in the correct format from the backup to the table. Add a manual pipeline in the CI/CD to trigger the execution of the script. The python script is changed according to the changes to the data model. 
- Problem is that the Python code does not have strict typing of the objects, difficult to check if it will write the right format to the table. The other problem is that the Python code does not make use of a shared spaceweather library, and duplicated code might become an issue. 
- Problem is that deployment times can grow out control

**Solution B**: Make our code backwards compatible.

Suggestion: Research ticket to learn about AWS migration. Check how dynamodb tables can be backed up.

Example: For the category we needed to extract extra information from the notification. It is possible to find the right notification by the UUID.

Story points: 3.0
Update Harmonie colours for layers also used in observations

Use the same colours as used in https://gitlab.com/KNMI/pub/rdwd/geoservices/-/blob/master/adaguc-config/includes/observation-styles.include.xml

Do this for:
- Temperature
- Pressure
- Precipitation

Make the WOW palette the default and keep the old as second one.

Change these files:
- https://gitlab.com/KNMI/pub/rdwd/geoservices/-/blob/master/adaguc-config/includes/StandardStyles_HARM_N25.xml
- https://gitlab.com/KNMI/geoweb/geoweb-adaguc-server/-/blob/master/adaguc-config/includes/StandardStyles_HARM_N25.xml

Use the ones on the left:
![image](/uploads/7281ec54655e2a2933bc77a61a888751/image.png)

Story points: 5.0
Presets: Default map cutout missing

When you click on the preset it is expected that the map is zoomed to the data belonging to that preset. If you use radar, it should zoom to the Netherlands. This should be fixed in the KNMI version.

Some of the presets in the KNMI GeoWeb version do not have a default map cutout:

* Radar (\*)
* Harmonie N25 wind and pressure (\*)
* Harmonie N25 temperature (\*)
* Satellite
* Observations (\*)

When using these presets, the map view the user is displaying will not be changed. This may lead to displaying data on a part of the map that is not visible to the user.

(\*) initial zoom level and center position of map can be the same as (e.g.) `current weather`

Story points: 2.0
FE:Theming p1:  Make reusable app layout

The layout is now a combination of div and css. We use react and MUI, we can use those for that. 
1. Create a component called AppLayout, put it in shared, and give it prop slots like 'header, content, footer'
2. The styling should only do the layout + zindex, no colors
3. Use it in knmi-geoweb, and the reference application

![Screenshot_2021-07-14_at_16.57.14](/uploads/a0a35a4a00bc8477c7cb5d4d2e0023a6/Screenshot_2021-07-14_at_16.57.14.png)

Story points: 5.0
FE: Theming p5: UserMenu styling

There are a couple of points need to be fixed:
- colors are wrong
- padding
- position dropdown
- lines/background color are wrong
- opacity should be inactive like design, and on hover as design
- dark mode

This ticket is a good try out for structure of theming. If this works correctly, then we pick up the other modules as well

| design | now |
| ------ | ------ |
| ![Screenshot_2021-07-13_at_15.02.27](/uploads/ee61c69e15ed95d54dd1f2e102e278da/Screenshot_2021-07-13_at_15.02.27.png) | ![Screenshot_2021-07-13_at_15.02.32](/uploads/a58f0a9a29c159cc892cdf6e2237547d/Screenshot_2021-07-13_at_15.02.32.png) |

Story points: 3.0
Check how SW summaries are handled

The MetOffice notifies us of summaries in 2 ways:
1. by re-sending the original notification, with a field `"summarized": true`
2. by sending a ..._ALERT_SUMMARY message, usually with some extra info about the event.

In the Sept. 2017 dataset and from our experience the last months one of those 2 messages may not arrive at KNMI or arrive a few days late.

This issue is about checking if this is handled correctly.
We might want to choose to ignore the first type of messages, but only if we're sure both types always arrive.

Story points: 5.0
Implement redux api (actions and reducers) for screenmanager

Implement a similar api like syncgroups and maps, but now for screenmanager.

Suggested is to have a similar structure in the redux state using allId's and byId to identify the screens.

Suggested actions:
- screenManagerAddView
- screenManagerRemoveView
- screenManagerUpdateViewCoordinates

The current POC for the screenmanager can then be updated to use the redux state instead of the local state.

Story points: 5.0
FE: statusTag is showing wrong text for a DRAFT summary or DRAFT cancellation

- issue a new proton flux alert
- open it and click summarise
- save as draft
- open it again and check the statusTag

expected: purple statusTag with text Summary

actual: purple statusTag with text Alert


NOTE: same issue for a DRAFT cancellation


![image](/uploads/828b8b63f72e8195410c598424dd89c2/image.png)

![image](/uploads/8a5a3302a78cdecd0f04fa9b6b36fb9e/image.png)

Story points: 3.0
FE: SW publish of Geomagnetic storm draft problem

* when retrieving a draft Geomagnetic storm watch the JSON received from the BE is correct however, when then publishing it, somehow the changestateto has changed to 'ended' instead of 'issued' which immediately cancels the issued watch. **This seems to only happen for the geomagnetic storm watch**. Other warnings seem to behave ok

Incoming draft:

```
{
  "eventid": "927ddf62-e07e-11eb-b77b-b186cd097fd0",
  "originator": "KNMI",
  "notacknowledged": false,
  "category": "GEOMAGNETIC",
  "categorydetail": "GEOMAGNETIC_STORM",
  "lifecycles": {
    "internalprovider": {
      "owner": "KNMI",
      "eventid": "927ddf62-e07e-11eb-b77b-b186cd097fd0",
      "eventend": "2021-07-09T09:26:00Z",
      "canbe": [
        "discard",
        "publish"
      ],
      "eventlevel": null,
      "draft": true,
      "firstissuetime": null,
      "lastissuetime": null,
      "state": "drafted",
      "label": "WARNING",
      "eventstart": "2021-07-09T06:26:00Z",
      "notifications": [
        {
          "neweventend": "2021-07-09T09:26:00Z",
          "changestateto": "issued",
          "neweventstart": "2021-07-09T06:26:00Z",
          "categorydetail": "GEOMAGNETIC_STORM",
          "draft": true,
          "label": "WARNING",
          "message": "dfgdfgdfg",
          "initialgscale": "600",
          "issuetime": "2021-07-09T06:26:19Z",
          "notificationid": "927ddf62-e07e-11eb-b77b-b186cd097fd0",
          "category": "GEOMAGNETIC"
        }
      ]
    },
    "externalprovider": null
  },
  "start_dt": "2021-07-09T06:26:19Z"
}
```

Then when publishing, this is published:

```
{
  "category": "GEOMAGNETIC",
  "categorydetail": "GEOMAGNETIC_STORM",
  "label": "WARNING",
  "initialgscale": "600",
  "neweventstart": "2021-07-09T06:26:00Z",
  "neweventend": "2021-07-09T09:26:00Z",
  "message": "dfgdfgdfg",
  "changestateto": "ended",
  "draft": false,
  "notificationid": "927ddf62-e07e-11eb-b77b-b186cd097fd0",
  "eventid": "927ddf62-e07e-11eb-b77b-b186cd097fd0",
  "issuetime": "2021-07-09T06:26:19Z"
}
```

you can see the changestateto has changed from `"changestateto": "issued",` to `"changestateto": "ended",`

*

Story points: 5.0
FE: SW title of dialog is lagging behind

* Probably not so important so can be left for a separate ticket. The title tends to lag behind sometimes. So let's say you've got an KNMI issued alert which you save as a draft. Then when reopening the dialog, the title says draft correctly. Then issue the draft, and re-open the same event. You see that now the title says draft! Go back and reopen, still draft. Go back, open a different event, close and re-open the first one, you see that only now it's updated to be correct!

Story points: 3.0
FE: Show spaceweather BE error messages in FE

If a message is sent from the BE we should show in the FE in the same manner we do as for aviation.

Currently the error messaging is also different - we show it in small red font above the publish buttons. Should we also make this similar to aviation by displaying a banner at the top of the dialog? 

We can use the banner at the top we use at sigmet/airmet

Story points: 3.0
FE: prepopulating new fields when summarising

Some of the new fields are only present when summarising, for example for a radio blackout the peak class/flux/time and end time fields. Currently they are empty by default. It would be good to prepopulate those with the values from the last external notification summary, if present.

From Eelco:
- When opening the form for editing (so after pressing the buttons) we should populate the fields with whatever has come from MetOffice and ignore what KNMI has changed (excepting of course the message field which should NEVER be copied over)

Story points: 3.0
Clean up Lambda's Metrics Logs for Spaceweather Lambdas

Clean up Lambda's Metrics Logs for Timeseries API, Notification API, Event Ingest, ReqX, ReqSW, ReqKP, ReqM, Bulletin, Notifications, Upstreams, Authorizer.. It should go there the following:

* START, END, Username, REPORT for every time Lambdas are called;
* Unmanaged Errors.

After the #941 and #942 everything should be already clean, so this ticket is pure cross-check!

Story points: 2.0
In Spaceweather BackEnd add the logs with INFO, WARNING and DEBUG log_level for SpaceWeather Lambdas.

In Spaceweather BackEnd add the logs with INFO, WARNING and DEBUG log_level for Timeseries API, Notification API, Event Ingest, ReqX, ReqSW, ReqKP, ReqM, Bulletin, Notifications, Upstreams, Authorizer.

Note: use WARNING and DEBUG only if there is something to be reported in those categories, otherwise leave it out.

Story points: 3.0
In Spaceweather BackEnd add the logs with ERROR log_level for Spaceweather Lambdas.

In Spaceweather BackEnd add the logs with ERROR log_level for Timeseries API, Notification API, Event Ingest, ReqX, ReqSW, ReqKP, ReqM, Bulletin, Notifications, Upstreams, Authorizer.

It is necessary to make the 'error try-catch' to all database and S3 operations (as it is done for Aviation).

Story points: 5.0
In Aviation BackEnd add the logs with INFO, WARNING and DEBUG log_level for Aviation Lambdas.

In Aviation BackEnd add the logs with INFO, WARNING and DEBUG log_level for Sigmet, Sigmet-TAC, Airmet, Airmet-TAC, Taf, Taf-TAC, Authorizer.

Note: use WARNING and DEBUG only if there is something to be reported in those categories, otherwise leave it out.

Story points: 5.0
Clean up Lambda's Metrics Logs for Aviation Lambdas

Clean up Lambda's Metrics Logs for Sigmet TAC, Airmet, Airmet-TAC, Taf, Taf-TAC, Authorizer.
It should go there the following:
- START, END, Username, REPORT for every time Lambdas are called; 
- Unmanaged Errors.

After the #939 and #940 everything should be already clean, so this ticket is pure cross-check!

Story points: 3.0
Meeting: Create master layerManager for multimap setup

Organise a meeting to discuss code structure/ideas.

This should only be used for multimaps where each map has the same layers (for example, for different model runs).

There should be one layerManager button that controls every map

This is required to complete https://gitlab.com/opengeoweb/geoweb-assets/-/issues/1531

Copy the Precipitation, pressure and wind preset and make a similar one but with one master layer manager, so it’s easy to compare both types of preset. Names can be `Harmonie precipitation, pressure and wind 1:1` and `Harmonie precipitation, pressure and wind 1:all` respectively 
- The master layer manager button can be in the first map for now

Story points: 5.0
FE: Add preset with different validtimes for a model

We will do it as part of the geoweb-knmi implementation.

In the componentsLookup, you have to add a new case, which will be similar to the HarmonyTempAndPrecip preset (that one also uses different reference times). 

Later on, we will create a master layer manager and master legend for this kind of setup.

Story points: 5.0
BE: Find out which IWXXM version we need

WMO is putting the IWXXM standards development in a GitHub repository: https://github.com/wmo-im/IWXXM

There is also a reasonably recent presentation about IWXXM versions at:
https://www.icao.int/MID/Documents/2021/IWXXM%20Implementation%20Webinar/IWXXM%20-%20PPT03%20-%20AI2%20-%20WMO%20What_is_IWXXM-final.pdf

Story points: 3.0
Monitoring: Exporting common functions for monitoring in utils (infra)

In Infrastructure BE folder are defined the same functions in each single lambdas file. Could be nice to export those functions in utils and make them available to be used when necessary:

The functions are:

* DecimalEncoder
* audit_log
* app_log
* err_log
* info_log
* warning_log
* debug_log
* get_lineno
* get_md5

NOTE: this development will be the same like the one for aviation https://gitlab.com/opengeoweb/opengeoweb/-/issues/1189

Story points: 3.0
Monitoring: Exporting common functions for monitoring in utils (spaceweather)

In Spaceweather folder are defined the same functions in each single lambdas file. Could be nice to export those functions in utils and make them available to be used when necessary:

The functions are:

* DecimalEncoder
* audit_log
* app_log
* err_log
* info_log
* warning_log
* debug_log
* get_lineno
* get_md5

NOTE: this development will be the same like the one for aviation https://gitlab.com/opengeoweb/opengeoweb/-/issues/1189

Story points: 3.0
FE: use react-sortablejs in layermanager

See https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/887 for inspiration
Add unit tests (see TAF for inspiration)

Also fix the drag issue with the hover of tooltips, as described here https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/942#note_706243747. This is happening on ALL tooltips

Story points: 3.0
BE: Monitoring: Fix the Infrasctructure Dashboard

An investigation is needed to understand the meaning and the working way of the following panels:
* API Gateway - Infrastructure API (This panel seems to be always empty)
* Token Lambda - Logs (This panel seems to be always empty)


**Fix the logstash filter to ensure the output from the token lambda and the api gateway end up in elastic search (and therefore the dashboard)**

Ensure that we can see the logging in the dashboard (needs an adjustment to ensure it uses the correct index). 


If necessary, create a follow-up ticket to edit the log messages

Story points: 3.0
Monitoring:  WARNING and INFO log_level for Aviation lambdas

Implement in Spaceweather BackEnd the logs with WARNING log_level for **reqX**, **reqSw**, **reqKP**, **Update Stream**, **Authorizer**.

Implement in Spaceweather BackEnd the logs with INFO log_level for **reqX**, **reqSw**, **reqKP**

Wait till common functions can be re-used from utils folder.

Story points: 1.0
Monitoring:  Exporting common functions for monitoring in utils (aviation)

In Aviation BE folder are defined the same functions in each single lambdas file.
Could be nice to export those functions in utils and make them available to be used when necessary:

The functions are:
- DecimalEncoder
- audit_log
- app_log
- err_log
- info_log
- warning_log
- debug_log
- get_lineno
- get_md5

Story points: 5.0
BE: SpaceWeather monitoring misses monitoring of error for Kp

In Spaceweather Dashboard (errors panel) is missed the triggering of alert for the errors of reqKp.

The monitoring of 'F' (logGroupName: "/aws/lambda/geoweb-spaceweather-reqkp" AND log_level: "ERROR") is missing.
It is necessary to add 

```OR sum () OF query (F, 5m, now) IS ABOVE 0```

Story points: 2.0
FE: [BUG] ScreenManager/synchronization groups: Zooming erases the timevalue of the timeslider (set back to 1970)

In the screenmanager application of knmi-geoweb: Zooming erases the timevalue of the timeslider (set back to 1970):

MapComponent fires both Area and Time actions, and only one of these is stored in redux state. These actions are overwriting each others state.

Solution: Update actions and selectors to make it possible to store the value for each action type (area and time) in the redux state

Story points: 5.0
FE: unit test taf fails sometimes

See for example https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1571779993 
and https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1571489878
https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1609192798
https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1607820958

Test should be stable

```
components/TafWrapper/TafWrapper › showing spinner when doing a post request to the backend › should show the spinner when publishing a taf from the dialog
310 |         expect(
311 |           container.parentElement.querySelector('[data-testid=contentDialog]'),
312 |         ).toBeFalsy();
```

Story points: 5.0
FE: unit test sigmet-airmet fails sometimes

See for example


https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1575582151 
and
https://gitlab.com/opengeoweb/opengeoweb/-/jobs/1575330370

Test should be stable

```
components/MetInfoWrapper/MetInfoWrapper › should open the dialog for a new airmet when creating a new airmet, save it and open a new empty airmet
144 |       expect(queryByTestId('productform-dialog')).toBeFalsy(),
```

Story points: 5.0
Fix issue monitoring Bulletin API failure.

On 7/09/2021 we got (continuously from 14 to 21) a strange alert from grafana related to an Error in one of our spaceweather lambdas (BulletinAPI) in PRD environment. 
It started at 14 when a new bulletin (containing 'error' in the message content) arrived and it ended around 21 when Ernst closed the SpaceWeather PRD application. 

After an investigation with CPT, the issue have been found in [logstash](https://gitlab.com/KNMI/cloud-platform/monitoring/logstash-config/-/blob/master/pipelines/knmi-geoweb/3002-filter-geoweb-backend.conf) 

The pattern `"%{GREEDYDATA:log_level} - %{GREEDYDATA:msg}"`  will make log_level match as much data as possible, in regular expression this is .*, until it finds the last  - . Instead it should only match 1 word: DEBUG, so it should be set to `%{WORD:log_level}`

Suggestions from CPT:
1. You can try it out for yourself at https://grokdebug.herokuapp.com/ and using `DEBUG - Successful: {"message": {"overview": {"type": "ISSUED", "simplified_content": "- example with dashes - test` as input. Then you see log_level will match much more, while if you use `^%{WORD:log_level} - %{GREEDYDATA:msg}$`  as pattern it doesn't. 
2. [not needed] better to use `^%{WORD:log_level} - %{GREEDYDATA:msg}$`. By adding ^ and $ to the beginning and end the pattern matching will be much easier to process for Logstash, because it knows this expression should not match something halfway the text

**In few words: the continuously alerting was caused by the word 'error' in the content of the message that triggered the error in grafana due to the use of `GREEDYDATA:log_level`.**

Story points: 2.0
Improve spaceweather bulletin-api logging - don't log entire bulletin text and history list

- Annoying to have the large messages in the logging
- not necessary to log the entire bulletin and entire historylist, this makes the logging hard to read

also see this [thread](https://knmi-nl.slack.com/archives/C01Q53A7U3U/p1631027510015700?thread_ts=1631019620.002400&cid=C01Q53A7U3U)

Solution:
- Only log some properties of the message, lambda should not output complete message. 
- Bulletin API, is triggered by a GET request
- Notification lambda, triggered two times per day

Story points: 3.0
Add cypress test for using the layer manager

In geoweb-serverless-backend:


**add smoketest for using the layer manager:**

login -> open layer manager -> check there is one layer row -> add another layer -> check there is 2 layer rows + 2 legends -> duplicate one layer -> check there is three layer rows + 3 legends -> remove one layer -> check there is 2 layer rows + 2 legends -> close the layer manager -> check that its closed

Story points: 3.0
FE: Use same CoreThemeProvider and CoreThemeStoreProvider instead of testutils

- Get rid of TestWrapper and TestWrapperWithStore in ./opengeoweb/libs/core/src/lib/utils/testUtils.tsx
- Use Providers from libs/core/src/lib/components/Providers/Providers.tsx
- Add DynamicModuleLoader to these providers and make test work (mock it: `store.addModules = jest.fn(); // mocking the dynamic module loader`)

Story points: 3.0
FE: update storybook packages to 6.3.7

update storybook packages to the latest version

note: all of them need the exact same version:
- @storybook/react@6.3.7 
- @storybook/addon-storyshots@6.3.7 
- @storybook/addon-storyshots-puppeteer@6.3.7

make sure that all storybooks and the story shot tests still work

Story points: 3.0
FE: update browsers list in package-lock

- run `npx browserslist@latest --update-db`

this will update the browsers in the package-lock file (used in the unit tests)

this will fix these comments popping up when running the tests locally

```
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db

Why you should do it regularly:
https://github.com/browserslist/browserslist#browsers-data-updating
```

Story points: 3.0
FE: theme and shared storybooks give console warnings for duplicate title + consistency naming

- nx storybook theme
- check the console:

5 warnings for Duplicate title: 'demo'

- nx storybook shared
- check the console:

![image](/uploads/858c8477ce449824ff7be88fd01db6ab/image.png)

- Also while you're at the titles, give the stories a consistent name. Some have `story` suffix, some `demo`

Story points: 2.0