giseldo/neodataset huggingface dataset:
InfoRefsUploadPack disk cache: file descriptors must be closed after use

RPC method **InfoRefsUploadPack** uses a special cache based on the use of files.  
Each requests tries to find a cache entry in it.  
If it exists the data would be read from it and returned as a result of the call.  
The read is done with use of `[os.Open(respPath)](https://gitlab.com/gitlab-org/gitaly/-/blob/v13.0.0-rc1/internal/cache/cachedb.go#L92)` and `[io.Copy(w, stream)](https://gitlab.com/gitlab-org/gitaly/-/blob/v13.0.0-rc1/internal/service/smarthttp/cache.go#L63)`.  
But the call to `stream.Close()` is missing and it results to file descriptors leak.  

/cc @zj-gitlab @jramsay

Story points: 1.0
Gitaly tmp folder cleanup must run every hour

Gitaly uses a special directory as a temporary folder named `tmp`.  
It has a [special background task](https://gitlab.com/search?group_id=&nav_source=navbar&project_id=2009901&repository_ref=v13.0.0-rc1&search=StartCleaning()&search_code=true&utf8=%E2%9C%93#L79) that must run each hour to clean up stale files in it.
But now it runs only once after start/restart because the call is not wrapped into the infinite loop.  
It could be observed by checking logs that a missed log entry for each hour.  
There are only entries after the service restart ![image](/uploads/eab41b2518b95f005e15271459f7cea6/image.png)

/cc @zj-gitlab @jramsay

Story points: 1.0
Update Source install instructions to use `make build-git`

[Source install documentation](https://docs.gitlab.com/ee/install/installation.html) includes instructions for installing Git. It has to be present to obtain the source, but needs to have the correct version too. 

### Proposal

Remove the Git install from the dependencies section: https://docs.gitlab.com/ee/install/installation.html#1-packages-and-dependencies, and move it to Gitaly to leverage `make download-git`.

Story points: 5.0
Frame-wise proxying of write operations to Gitaly replicas

## Problem to solve

When a success signal is sent via the Git protocol or an RPC for a write operation from a HA Gitaly cluster, we want the success signal to communicate that the write has succeeded in multiple locations so that a single node failure does not result in data loss.

Write operations are currently routed only to the primary node. This data also needs to be sent to the other nodes in the cluster.

## Further details

Replaying the write operation from the primary to the secondary nodes is not ideal from a performance or reliability perspective.

It should be possible to proxy the write to all the Gitaly nodes. The simplest approach is frame-wise proxying, but if one node is slow, this will block all the nodes. In the future, stream-wise proxying would solve this.

## Proposal

Implement frame-wise proxying of write operations to all Gitaly nodes as part of strong consistency.

## Testing

Functional end-to-end coverage is provided by an existing [replication test](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27972). And broad coverage by [running the entire end-to-end test suite against environments using Praefect for storage](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationpraefect-ceeefull-image-address) (including Staging).

Performance end-to-end testing will be implemented as part of https://gitlab.com/gitlab-org/quality/performance/-/issues/231 (see https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/451 for a more detailed plan of performance tests under failure conditions)

## Links / references

Story points: 3.0
Error in go FetchInternalRemote "git fetch failed: signal: terminated"

`FetchInternalRemote` sometimes fails with `git fetch` getting SIGTERM. And returning an error as per https://github.com/golang/go/blob/go1.14.2/src/os/exec_posix.go#L107

https://log.gprd.gitlab.net/app/kibana#/doc/AW5F1OHTiGcMMNRn84Di/pubsub-gitaly-inf-gprd-001969?id=wJYRx3EB_vQvi1G70XPU

Story points: 3.0
Praefect added latency misjudging CalculateChecksum latency

When looking at the added latency through Praefect, it seems like only CalculateChecksum has issues, which seems odd:

![Screenshot_from_2020-04-28_13-58-50](/uploads/70ba78a22284b7f473fe04a6f2e5a669/Screenshot_from_2020-04-28_13-58-50.png)

Checksums are used extensively by Praefect, without proxying the RPC to a client. It seems like a bug in the metric.

Story points: 2.0
Praefect: proper multi-virtual storage support

Praefect has advantage of multi-virtual storages support in configuration, but in the [real life](https://gitlab.com/gitlab-org/gitaly/-/blob/ca7faa446b4db4713ad128698d4130aa4de8a26e/cmd/praefect/main.go#L250) it can't handle it properly.  
Current routing logic doesn't support multi-virtual storage configuration and allows request routings only for single virtual storage. As well replication is also has no multi-virtual storage support, that affects a helper-subcommand: `dataloss` as well.  
Proper multi-virtual storages support is required before https://gitlab.com/gitlab-org/gitaly/-/issues/2650

/cc @jramsay @zj-gitlab

Story points: 3.0
Revise usage of ReplicasDatastore interface

After moving to new approach with leader election based on different strategies most of the methods of the `ReplicasDatastore` interface are not needed anymore.  
They exist only because of the tests.  
To simplify project we need to clean up and refactor it as it could lead to misunderstanding of proper usage.


/cc @zj-gitlab @jramsay

Story points: 2.0
The `ReplicateRepository` GRPC should automatically clean up after itself when it encounters an error

The `ReplicateRepository` GRPC should automatically clean up after itself when it encounters an error.

Currently, errors appear to get sort of eaten, and the invoking Rails process never notified about the failure.

This results in a bad state, with a project repository only partially replicated to the destination gitaly storage shard file system.

In any case, once the underlying problem is fixed, that is all well and good.

However, to future-proof against other unknown and as yet unencountered errors, the `ReplicateRepository` GRPC implementation should include an unwind/rollback step which is ensured to be invoked when any error is encountered.

This would also significantly unblock our current process, and would allow us to get back into the business of spinning through large numbers of project replications as part of storage shard re-balancing operations.

The failed replica repo should never matter, since the original repository should remain untouched, as defined by the scope of the `ReplicateRepository` GRPC method implementation.

Story points: 3.0
Enable SQL leader election by default

SQL leader election https://gitlab.com/gitlab-org/gitaly/-/issues/2547 is available, but is not enabled by default. SQL based leader election should be the default behavior, even if automatic failover is not enabled.

## Proposal

When not configured the default leader election strategy should be `sql`:

```ruby
praefect['failover_election_strategy'] = 'sql'
```

## Links / references

Story points: 1.0
Praefect behavior when replication jobs fail to persist

Currently, Praefect will allow a proxied RPC to succeed even if the associated replication jobs fail to persist. We simply log any replication persisting error and it doesn't impact the success/fail message to the client. This is incorrect behavior. When Praefect proxies an RPC, it is promising the end user a number of things:

- The RPC will be proxied to the primary.
- The replication jobs from the primary to the secondaries will be scheduled.

If the replication jobs fail to be scheduled, we should fail the RPC since we did not satisfy the contract to the end user.

Story points: 2.0
FE: use React 18

update packages to use react 18:
- react
- react-dom
- react-test-renderer
- @testing-library/react": "^12.1.2", -> 13
- @testing-library/react-hooks": "^7.0.2", -> uninstall and import renderHook from above package
- @types/react
- @types/react-dom

Make sure this branch is uptodate with master.

If possible this ticket should be splitted into smaller ones. Fix upcoming react changes separately (if possible)

TODO:

- [x] wmjsmap doesn't currently work with React 18 in Strict mode, this has to be fixed. The map view gets mounted, unmounted and not remounted. We could temporarily disable strict mode for the app until it's fixed. More info: https://reactjs.org/blog/2022/03/08/react-18-upgrade-guide.html#updates-to-strict-mode. Fix in separate ticket https://gitlab.com/opengeoweb/opengeoweb/-/issues/2957
- [x] typing in a TAF field breaks the app, try to update react-hook-form to V7 first? https://gitlab.com/opengeoweb/opengeoweb/-/issues/2926 Not sure if this would fix it, maybe needs a bit more investigation.
- [x] fix tests (problems with `act` warnings in tests: https://github.com/testing-library/react-testing-library/issues/1051)
- [x] screenmanager issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3093
- [x] login issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3094
- [x] drag and drop issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3095
- [x] map presets issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3100
- [x] spaceweather graphs issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3101
- [x] spaceweather notification populate issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3102
- [x] sigmet level values issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3103
- [x] TAF autosave issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3180
- [x] Sigmet Volcanic Ash Cloud issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3181
- [x] Airmet Cloud levels issue https://gitlab.com/opengeoweb/opengeoweb/-/issues/3182

Story points: 8.0
FE: make ContentDialog responsive

DEsigns can be found https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/5f4e0b5cef6167b007682172

- (long) title should look correct on small screens
- action buttons should get optionlist menu 
- don't use javascript to get it completely like design in viewport of the dialog. We should wait for container queries 
- make sure the form elements look correct

![Screenshot_2022-10-13_at_14.53.30](/uploads/3bab33bd95e63df9172bea2fd060c75c/Screenshot_2022-10-13_at_14.53.30.png)

Story points: 2.0
FE: fix storybook warnings

Some stories generate warnings, we need to change to the new syntax. Also update the readme (for the zeplin links) where needed.

![Screenshot_2022-10-13_at_14.42.06](/uploads/621c86c2551b32ab01d329c87ba53ff0/Screenshot_2022-10-13_at_14.42.06.png)


In 5.x CSF, you would annotate a story like this:
```
export const Basic = () => <Button />
Basic.story = {
  name: 'foo',
  parameters: { ... },
  decorators: [ ... ],
};
```
In 6.0 CSF this becomes:
```
export const Basic = () => <Button />
Basic.storyName = 'foo';
Basic.parameters = { ... };
Basic.decorators = [ ... ];
```

Try to run this codemod: `npx @storybook/cli@next migrate csf-hoist-story-annotations --glob="**/*.stories.js"`


see https://github.com/storybookjs/storybook/blob/next/MIGRATION.md#hoisted-csf-annotations
expected result: no more warnings in any story or snapshot test

Story points: 2.0
FE: sigmet-airmet remove tooltip after clicking on list element

As pointed out in https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1920#note_1134117242, when you hover on a tooltip, and open a sigmet/airmet it will briefly show the tooltip in the dialog. It was already there, but it's more visible now we have a bigger tooltip. 

When opening a sigmet/airmet it should close the tooltip so it won't be visible inside the dialog. Maybe easy fix on zIndex? Should be an easy fix, otherwise we have to check other options

Maybe it will be fixed automaticely in https://gitlab.com/opengeoweb/opengeoweb/-/issues/2790 by the pointer-events, do that ticket first to check

Story points: 3.0
Sigmet/airmet move TAC tooltip from i icon to issue time

Current situation is Tooltip of TAC is on the `i` icon, but in design it has it on issue date. If we do this, we save a column, but that is impacting the [responsive ticket.](https://gitlab.com/opengeoweb/opengeoweb/-/issues/2753) 

- question for design, any reason not to use the default tooltip color?

![Screenshot_2022-10-11_at_15.06.22](/uploads/72765a01785120a8730b0262b9f6987d/Screenshot_2022-10-11_at_15.06.22.png)

Story points: 2.0
BE: investigate AWS SAM to run BE locally

Investigate and document on whether we can use SAM to run the lambdas locally for easier testing.

Add documentation on how to use and set up

Findings:
- SAM templates are a superset of CloudFormation templates
- SAM templates can include "transforms" that could make the definition of serverless apps more breve
- The SAM CLI has a `sam local` command that make it possible to run your stack locally, but the number of supported services is very limited (i.e. no S3 support)
- Because of SAM's limited support I had to include LocalStack for S3 emulation
- This makes it impossible to just re-use existing CloudFormation templates, so I ended up creating new CloudFormation templates just for development, combined with Bash scripts
- LocalStack has emulation for some services as well, so I tried that too, without using SAM. CloudFormation support is very limited, so again I found myself creating a new CloudFormation template just for development, combined with Bash scripts
- It is a painful process and I didn't get a full stack working within one day - I'll stop my investigation on this now and first discuss with the backend people of the team

Story points: 5.0
FE: make layer select responsive

With the dimension icons added, the layer row needs to get more height in the minimum size to display everything. 

- update service chip height to match design
- make sure layer row is responsive

see design
https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/62c695a1ed42f01adee34e63

![image](/uploads/9a8f5b945a6793c02ecf0903107a8aa7/image.png)

Story points: 3.0
FE: theming sigmet/airmet part 3: Form

Fix the last part; the Form. All OldThemeWrapper references should be removed from the sigmet/airmet lib. Change it as well in the geoweb application.

See how big this ticket is, still things to fix in this ticket or as follow up:
- remove white background of form
- enable commented out test
- remove all `oldTheme` props

Story points: 3.0
FE: theming sigmet/airmet part 2: ContentDialog

Add styling for the ContentDialog. This is the outer dialog, so wrap the form with OldThemeProvider so that will looks the same. 

- also add links to design

Style everything outside the red line:
![Screenshot_2022-10-10_at_10.16.40](/uploads/3ef650ecb3a0114a655347fe065d7700/Screenshot_2022-10-10_at_10.16.40.png)

Story points: 2.0
FE: theming sigmet/airmet part 1: List component

Add StoryWrapperOldTheme wrapper to the contentDialog so that will still show the old theme. Fix the theming of the List component so it will have dark/light theme.

- also add links to design
- check responsiveness if it's not too much, otherwise create a separate ticket for it

![Screenshot_2022-10-10_at_10.13.08](/uploads/8a4efc4f6e0e5eec33455a8820d80e26/Screenshot_2022-10-10_at_10.13.08.png)

Story points: 5.0
FE: show dimension icons in layer select rows

- show elevation, time and reference time icons in layer select rows when the layer has the dimension
- There should be a hoover function showing
  * the name of the dimension
  * the value
- 'other dimensions' are for the next ticket

![image](/uploads/746f35dccdca3f3ea8e316bf3e57aee9/image.png)

Story points: 2.0
FE: move FE publish to npm jobs to the tag pipeline so they can run automatically

The "publish package to npm" jobs have to be started manually now on the master branch when a new release was created. This is some leftover debt from when we started the project and didn't have tags or docker images.

We should move them to the tag pipeline so they can run automatically.


(all of these except the `docker-images` job)

![image](/uploads/ad5fa16e96b54e11237483375d106494/image.png)

Story points: 2.0
FE: Fix flaky tests

- some tests that fail on a millisecond difference can be fixed by mocking the date
- investigate other tests and fix




- list of tests that are currently flaky:
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3135758738
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3149521049
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3250367760
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3393741234
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3480513688
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3500579718
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3502784054
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3513811196
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3542313295
  - https://gitlab.com/opengeoweb/opengeoweb/-/jobs/3546948022
  -

Story points: 3.0
Show 'Other dimensions' in LayerInfoDialog

a layer can have multiple other dimensions. For example 'ensemble_member', 'modellevel', 'member'. Should each 'other dimension' get an icon? or altogether displayed under 'other dimensions'? Sorted alphabetically or not?

-> each 'other dimension' gets the same icon
-> show all values, even if its big
-> instead of text 'Other dimensions' show the name of the dimension
-> show to Didier how it looks

![image](/uploads/ed127297e41268ba0918a33ff450fa86/image.png)

Story points: 3.0
Show Dimensions and Bounding Box in LayerInfoDialog

in the previous ticket https://gitlab.com/opengeoweb/opengeoweb/-/issues/2711 the dimensions and geographic bounding box information is saved in the ServiceLayer object. Now we can use this in the LayerInfoDialog to show the information.

![image](/uploads/70b8dacc94dec474d57f6c7bb7f7ee91/image.png)

Story points: 2.0
FE: Clicking twice to make layer select buttons work if another dialog is ordered on top

see thread here https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1874#note_1121826230

fix for the info buttons, the add/remove layer buttons and the clear search button

it currently works for the layer manager buttons and layer select filter chips

Please make sure it is a nice at the source of the problem, discuss if solution is too big or too weird

Story points: 3.0
Convert DialogType strings to enum DialogTypes

in `opengeoweb/libs/core/src/lib/store/ui/types.ts` there is a type DialogType with string values. Convert this to enum, like is done for 'layerInfo'. Then you can replace the string values, for example `'layerInfo'` with `DialogTypes.LayerInfo` when firing actions and in tests.

```
export enum DialogTypes {
  LayerInfo = 'layerInfo',
}

export type DialogType =
  | 'layerManager'
  | 'layerSelect'
  | DialogTypes.LayerInfo
  | 'dimensionSelect-elevation'
  | 'dimensionSelect-ensemble_member'
  | 'keywordFilter'
  | 'timeSeriesManager'
  | 'timeSeriesSelect'
  | 'syncGroups'
  | LegendDialogType;

Story points: 3.0
use useSetupDialog hook in all dialogs

All dialogs use the same way to set up. This causes a lot of duplicate code. Instead we can use the same hook in all these dialogs. The hook already exists but is only used in the timeseries lib: https://gitlab.com/opengeoweb/opengeoweb/-/blob/9c4546ee9a2f57f0e142daf3ca62a811fba01774/libs/timeseries/src/lib/hooks/useSetupDialog/useSetupDialog.ts#L26

To Do:
- Move useSetupDialog to core folder. It can't be moved to shared folder because it will cause circular dependencies
- use useSetupDialog in all dialogs
- useSetupDialog is unit tested. For all the dialogs that start using useSetupDialog remove the tests related to what useSetupDialog does.

Story points: 3.0
FE: Update form-fields TextField with isReadOnly prop and use it in LayeSelect

Add a isReadOnly prop to all inputs of form-fields. If it's used, it will show the correct styling. 

- add prop isReadOnly for all inputs. Make sure it will also work with disabled also as prop
- when prop is there, add styling
- above the styling add a comment that it will be fixed in v6 https://github.com/mui/material-ui/pull/32822 
- update LayerSelect, if too big split it in separate ticket 


extra info
```
<TextField
InputProps={{
              readOnly: true,
            }}`
```

![Screenshot_2022-09-30_at_07.49.34](/uploads/30563806cdb91e68df1dc60bd9da35a1/Screenshot_2022-09-30_at_07.49.34.png)

The styling should be updated with latest design. https://zpl.io/29xdAPW


- in theme add a new TextField in the story with readonly prop, and style it like design
- after this, also add a TextField with readonly + disabled, so we can't focus. See if we can make it work together, but make sure this won't break the normal disabled styling, this should be unaffected. So if both props are present, it should still look like readonly, but without focus. 
- update the links to design in theme storybook for textField (link to correct version, light and dark theme)

This should fix the styling in the ServicePopup, so that snapshot should get an update. No code changes should be needed there, perhaps only removal of `disableUnderline: isShowPopup,` that might not be needed anymore

Story points: 3.0
Follow up info service: add dark/light theme snapshots for service dialog

Follow up ticket for https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1856 snapshots as discussed in MR here https://gitlab.com/opengeoweb/opengeoweb/-/merge_requests/1856#note_1116199475

Can you add dark and light stories with snapshots and link to [design](https://app.zeplin.io/project/5ecf84a3c6ae1047a368f393/screen/62c695a1ed42f01adee34e63/version/6332a6d892034711d79d6233) for the System service dialog? And then you can easily see that is does not match the design, so styling for readonly fields needs to be updated as well.

Story points: 2.0
Investigate momentjs alternative

Create a POC for a new library replacement for momentjs. See https://momentjs.com/docs/#/-project-status/ for alternatives. Make sure the lib is compatible with MUI. In the lib `form-fields` we are using [an adapter](https://gitlab.com/opengeoweb/opengeoweb/-/blob/master/libs/form-fields/src/lib/components/ReactHookFormProvider.tsx#L20) for the DatePickers, that should still work. In the POC, just compare how some basic date utils work like
- utc
- compare time between dates
- make sure the datepickers are still working

Story points: 3.0
Add new "icon" variant for buttons

The info button in the layer manager does not have space for a big mouseover background.

Make a new "icon" variant for the CustomToggleButton that can handle active state, the background on mouseover is hidden and the colors match design.

Design will be updated

![image](/uploads/f1e1ab8a3a6bdca7a58254cfd8745ee1/image.png)

Story points: 2.0
Update edit icon

Edit icon already exists but it needs to be updated.
Discussed with Didier as this icon is already used in multiple other libs, it should be updated everywhere.

TODO:
- update new edit icon path:
```
<path
        d="m12.718 3.807-5.214 9.426-1.3 2.325.325 5.58 4.905-2.683 6.494-11.711-5.21-2.937zM11.514 5.98l-.616 1.111 5.23 2.9.616-1.11-5.23-2.901zm-4.69 8.462-.612 1.113 5.225 2.898.616-1.11-5.23-2.901z"
        fillRule="evenodd"
      />
```
- update snapshots

Story points: 2.0
BE: POC: Use FastAPI for presets-backend

See user story for description.

Findings:
- FastAPI and Flask have many similarities, so it isn't to hard to switch form Flask to FastAPI
- FastAPI supports asynchronous calls, which makes it potentially faster (in our case we probably would have to use a DB driver that supports async calls to benefit from this)
- FastAPI requires you to define the structure of incoming/outgoing data and automatically applies validation
- FastAPI automatically generates OpenAPI specs and Swagger dashboard from code
- I spent 2,5 days on presets-backend and it's almost done (probably one more day for the last details)

Story points: 5.0