giseldo/neodataset huggingface dataset:
EPF: Define Useful DataTypes for Application

### Problem to Solve

Currently we have just a pile of quick-and-dirty created DataTypes as for a first draft

We need the some DataTypes, which can be used by EPF, Backend and Frontend.

Now we planned them as single words, but of course we could design them as slug+name,
e.g. "image" and "Image" is quite the same for single words.
"satellite-data" and "Satellite Data" may differ, but is easy to calculate the relation between slug <-> name

### Technical Solution

Data Types suggested

The data types can be grouped in these classes:
- Tabular: csv files, numerical, categorical, usually each row represents a sample, each column the features.
- Time series: csv files in which each row has time information , the rows are ordered usually by time. In this case (medical signals, economic indicators, prices of goods, trajectory of a car...)
- Images: the images(png,bmp, jpg..) usually come in folders with the class name, or a csv files with images in string form.
- Voice:  Recordings mp3, wav, usually come in folders with the class name.
- Video:  Recording  wmv, mp4, mkv, usually come in folders with the class name.
- Hierarchical (optional it can be considered a subcategory of tabular): Json files that represent data and its hierarchy, usually can be considered as tabular data. for example, positional data, landmarks for face recognition, tags of objects inside an image.. etc.

Output data types 
- model : model saved as a binary file.
- All the input datatypes apply here as well.

Story points: 1.0
Backend: Add publication fields to DataProcessor/MarketplaceEntry

### Problem to solve


### Technical Solution

Add to DataProcessor or MarketplaceEntry:


* [x] "published_at": Timestamp? 
* [x] "entrypoint":String (with file ending implicitly)
* [x] "base_environment" either as slug which refers to the enum class
* [x] "accepted_at" and "accepted_by" as time and String/ or Subject_id

Additional Notes
=====================

* Support a base environment text id ("slug") in the backend for DataProcessor
* Add entrypoint/command string path to Dataprocessor
* A dataProcessor Dto has a "is_published" flag(available via MarketplaceEntry Link)
* backend: accepted_at, or accepted_by as fields for legal reasons AND version of publishing terms

Story points: 4.0
Backend: Prepare DataProcessor publication

### Problem to Solve

The backend shall offer the users/frontend and endpoint to  publish/release a DataProcessor

* some quality measuresments can be required: 
  * name and description set
  * input and output dataType defined
* a MarketplaceEntry is created

Think how the annotations and paramters will be translated and saved into backend, possible another endpoint triggered by EPF

### Technical Solution

* [ ] endpoint with minimum input: global_slug, predefined Tags

We decided that estimation is just for the minimal part, as we dont now anything about EPF


Unsure:
* [ ] creates and starts the magic gitlab release pipeline




Story points: 5.0
Reset password

### Problem to Solve

As a User I want to be able to reset my password, if i forgot it.

### Technical Solution
The technical solution will depend on the backend implementation

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
Link to XD: https://xd.adobe.com/view/3d6242a2-2056-4eb2-5494-ca251026019c-4b53/

![Reset_pass](/uploads/72d2d235e898e77621069e366f071a39/Reset_pass.png)

![Reset_pass_1](/uploads/06ecdf7176faea5f9052a4bcf3446a6a/Reset_pass_1.png)

![Reset_pass_2](/uploads/7e1618ccbc96bd3c95b2773a356f9ede/Reset_pass_2.png)


blocked by: #527

Story points: 11.0
Create Impressum, Terms and Release pages on website

### Problem to Solve

Create the views for 
* [x]  Impressum
* [x]  Terms of use
* [x]  Release terms

based on the current texts of mlreef.com and the XD design template

### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
Link to XD: https://xd.adobe.com/view/7324878f-065f-4544-43b4-35bb877717e6-0515/

Story points: 1.0
Backend: Experiment's input files must be saved and ONE File must be stored in mlreef.yml

### Problem to Solve
The files selected for an experiment are not yet being stored in the backend and hence is resulting in breaking up of certain UI components.
https://gitlab.com/mlreef/frontend/uploads/883171798cc826c18914d704f3beddeb/image.png

The information about the files on which an experiment has to be run is not stored within the config file `.mlreef.yml` as well.


### Technical Solution

Frontend can send a list of files.
Backend can store a list of files.

But generating the mlreef.yml makes only sense with ONE path (which is usually a folder).

So, when generating a .mlreef.yml we take just ONE path (the first of the saved requests

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 3.0
Populate Data Visualization in Backend

### Problem to Solve
Currently, there are no Visualizations available in the backend.

![visual](/uploads/015bcc6e9715ae0f877c38d15b8bd1bc/visual.PNG)


### Technical Solution
* [x] waiting: ensure the are python files which can run as visualisation dataProcessor
* [x] @erika.torres please help backend to define that dataVisualisation
* [x] real effort: create that in dataPopulator

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 2.0
Adapt navbar to not registered useres (Visitors)

### Problem to Solve

When a User is not signed in but is exploring content on MLReef (such as an open repository), the navbar should be the "logged out" navbar. 




### Technical Solution
> **Optional:** Add technical implementation details and the results of the ticket's discussion here.

Navbar should be fixed (not scrolling). 


### Terminology



Additional Notes
=====================

Link to XD - **first slide!**: https://xd.adobe.com/view/c6c402fa-d7f8-4b46-4c6f-dd6257402fb4-b0f5/

![Data_Rep_Overview_A_loading](/uploads/8db91a06ec142ae2f9f6d1d6a53ebb66/Data_Rep_Overview_A_loading.png)

Story points: 6.0
Rename functions in Frontend

### Problem to Solve

We need to rename several functions in the frontend: 

* [ ]  Data Pipeline - DataOps
* [ ]  Data Instance - Datasets
* [ ]  Parameters advanced - optional
* [ ]  Market place "own" - "my ML projects"...
* [ ]  Data types in project creation (@Erika takes care of this list)
* [ ]  publishing - releasing
* [ ]  Publication - releases
* [ ]  Experiment pipeline, Algo 1 - Model 1


### Technical Solution
> **Optional:** Add technical implementation details and the results of the ticket's discussion here.


### Terminology



Additional Notes
=====================

Story points: 1.0
Group detailed view

### Problem to Solve

As a User I want to see what projects a group owns. This is done via the groups detail view, which can be accessed by clicking on a group.


### Technical Solution
> **Optional:** Add technical implementation details and the results of the ticket's discussion here.


### Terminology



Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/4cec0111-f89b-4825-6363-58d95e361fef-3add/

![Group_overview_projects___1](/uploads/bfdf9587b1a70403b46eea63c2e7004a/Group_overview_projects___1.png)

![Group_overview_projects](/uploads/0ff25676010e79c77a4f576e51a7d369/Group_overview_projects.png)

Story points: 4.0
Branch dropdown shows wrong branch - shows always master

### Observed behaviour

When switching to a different branch, the branch dropdown shows always the "master" branch. 

### Expected behaviour

It should show the selected branch, as this is the only way to know in which branch you are.


### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
The branch is probably hardcoded or it's not updated after selecting one in the dropdown.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/c2c33db3f48baed3a55a70355e9251d8/image.png)

Story points: 2.0
Write import data API requirements

### Problem to Solve

For ESA BIC we need to write and demonstrate that we did! the requirements for our data import API. We currently plan to integrate with 
www.sinergise.com and radiant.earth (mlhub). EODC has agreed to integrate as well, but I am not sure on how interested they really are. Further platforms and data archives will follow.

### Technical Solution

Lets write a new requirements.md in the company repo. 


### Terminology



Additional Notes
=====================

API documentation:
* http://docs.mlhub.earth/#radiant-mlhub-api
* https://github.com/radiantearth/stac-api-spec



Radiant Earth is STACK (Spacial Temporal … ) API compliant definition (REST API) does not include file size right now.

JSON with metadata including links to the actual assets.

API follows that stack and on top of that they build a proxy for additional business logic - they can add any kind of additional functionality that we need on top of that.
STACK enables filtering and searching on a metadata level.

Sentinel Hub also deploys a STACK- compliant API

Authentication is needed: Simple Self Service Signup with an API key.

How are users going to use the data and derive value from it?

Licensing is generally open and for public use

### light datasets
In regards to User-Experience, it would be a possibility would be to limit the users to smaller datasets.

Story points: 5.0
Create code repository views

### Problem to Solve

Code repositories are very similar to data-repositories. These are the main differences: 

*  Code Repos do NOT have data pipelines (visualizations, dataops, experiments)
*  Code Repos have a publishing process (described in frontend#447)
*  Code Repos do not have GIT-LFS as a standard enabled

All other features currently present in our data repository can be used in code repositories.

There are currently three types of code repositories which are also handled seperately in the backend: 
1.  Models
2.  DataOps
3.  Data Visualizations

The only visual difference between them is the color of the global marker and the naming of functions and buttons (e.g. "publish model" instead of "publish DataOps". 

### Technical Solution
> **Optional:** Add technical implementation details and the results of the ticket's discussion here.


### Terminology



Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/0743573d-9d17-4819-6dd4-fb5bc8fd63cc-7172/

![Data_Rep_Overview_A___1](/uploads/380881fbfe4465c35214369b1e2a78f3/Data_Rep_Overview_A___1.png)

![Data_Rep_Overview_A___2](/uploads/5be3a24fe4f577d9fa5879b36d68c231/Data_Rep_Overview_A___2.png)

![Data_Rep_Overview_A___3](/uploads/eb80cff7bdf5cd12ac81f9b9b9247866/Data_Rep_Overview_A___3.png)

![Data_Rep_Overview_A](/uploads/d37a4c7b4c63390a7ef807da17b85599/Data_Rep_Overview_A.png)

Story points: 8.0
ML Project / Insights / Storage

### Problem to Solve

As a User I want to see how much storage my ML project ocupies. 

For now, only the total storage will be displayed. Later, we can think about breaking it down into smaller pieces, such as LFS part, artifacts, etc. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/98e48030-fed8-42e8-5b6a-3dfeb3ee0906-eed5/

![Insights_Storage___1](/uploads/79d58863f101e4ddf828bec2cd2077ad/Insights_Storage___1.png)

![Insights_Storage](/uploads/fa143121def17d14b20538a881360d1e/Insights_Storage.png)

Story points: 6.0
Move Data Pipeline Creation to the Backend

### Problem to Solve
In order to correctly hand over management of Data- and Visualization Pipelines to the backend the Frontend has to use the Backend's API endpoints instead of directly interacting with Gitlab.

This includes 
* Creation, and starting of Data- and Visualization Pipeline
* Cancelling Data- and Visualization Pipelines
* Displaying the status of pipelines

In connection with this, the current API code and all related entries in the `dataTypes.js` shall be removed.

### Technical Solution
There exists a backend endpoint for data pipelines which will
create a new branch in the Git Repository
configure the .mlreef.yml for the User’s pipeline
starts the Pipeline in Gitlab.

* [x]  remove `adjectives` and `nouns` arrays from `dataTypes.js`
* [x]  Add at least one unit test
* [x]  If useful: Add documentation to the readme or the code

Maybe we should links to the existing endpoints
* https://mlreef.gitlab.io/backend/develop/#pipelines
* https://mlreef.gitlab.io/backend/develop/#pipelineinstances


### Terminology


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 6.0
Use backend's Data Processors API Endpoint

### Problem to Solve
Currently, the Frontend is rendering Data Processors (options for Experiment, Data Operations, and Visualization pipelines) based on the mock data in `dataTypes.js`.


### Technical Solution
In order to add more and also user-generated Data processors, the Frontend shall use the backend's API endpoint

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme or the code


Additional Notes
=====================
/cc @flohin

Story points: 3.0
Add endpoint for cancelling Experiments

### Problem to Solve
In order for the Frontend to cancel experiments, the Backend must provide an appropriate management Endpoint


### Technical Solution
**This endpoint shall be an Idempotent endpoint**

The request is proxied through the backend because, 

* BE receives cancel request
* passes it on to Gitlab
  * if the job is already not running, update the Backend database
  * if cancel was successful => save the new status in the Backend's database
* respond to frontend


* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme or the code


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 4.0
Backend: Add outputPath to generation of .mlreef.yml

### Problem to Solve
In order for the EPF to be able to correctly store a data operation's output in a certain folder, the Backend has to create the `.mlreef.yml` correctly

### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.

* [ ]  Add at least one unit test



Additional Notes
=====================
duplicates: #507

see: #319

Story points: 2.0
Remove mock data from Frontend

### Problem to Solve
It is time to get rid of the mock data in the frontend and actually use the existing functionalities of the backend.

Parts of `dataTypes.js` which will be refactored by in other Tasks

### Technical Solution
Get rid of the mocked data `dataTypes.js`

Move actual configurations to `apiConfig.js` and request endpoints for still missing data from the backend team.

* experiments -> #265 
* dataVisuzalizatons Array -> #458 
* dataPipelines Array -> #458 
* adjectives and nouns arrays -> #459
* [ ]  enumeration pipeline statuses: can maybe stay?
* [x]  `filesForExperimentsDetails`?
* [ ]  `/* -------------------------- Project classifications -------------------------- */` (probably can be deleted)
* [ ] `privacyLevelsArr`?

Story points: 2.0
Feature: Backend handles gitlab ci and pipeline starting

### Problem to Solve

Necessary for experiment and dataPipelines

see #265 

Will adapt the model:
* store GitlabPipeline info in backend (new req)
* rename metrics timestamps to gitlab pipeline info timestamps
* store local id of experiment (1 for first) as "order" or "number"
* store gitlab_pipeline_id into experiment
* fix migrations again
* fix tests

### Technical Solution

* Write gitlab ci tests
* add more gitlab client requests for starting pipelines and handle ci files

Story points: 4.0
Find how to upload image files to Giltab

### Problem to Solve
The GitLab API currently does not allow to upload images. The cause of the bug is not obvious since the documentation is really poor about this functionality, it just hints that multiple formats are accepted.

Find the way or format in which Gitlab can process image files so that we can change avatars or upload files to repositories.


### Technical Solution

### Terminology
Gitlab API, Image format


Additional Notes
=====================
https://docs.gitlab.com/ee/api/groups.html#new-group (see the avatar field in new group payload).

Story points: 4.0
Make Deploy script restartable

### Problem to Solve
Sometimes, the deploy script fails for temporary reasons. E.g: the docker registry timed out. If this happens after _Step 2_ (Gitlab port configuration). The script cannot just be restarted.


### Technical Solution
There are different solutions possible. Either "surgically" remove configurations which can lead to the script failing on a second run (e.g. Gitlab's `external_url`)

or

Add additional `if` or `goto` statements which cover for a Gitlab that has already been configured


Additional Notes
=====================
It happened in this [pipeline](https://gitlab.com/mlreef/frontend/-/jobs/521324252)

In this the [first job](https://gitlab.com/mlreef/frontend/-/jobs/521311679), during the last `docker-compose pull` the registry timed out and the script failed 

In the [second job](https://gitlab.com/mlreef/frontend/-/jobs/521324252) the script was waiting for Gitlab at port 10081.  BUT Gitlab had already been reconfigured to the final port 10080

Story points: 5.0
Add instructions for contribute with documentation

The following discussion from !407 should be addressed:

- [ ] @cpmlreef commented on a [discussion](https://gitlab.com/mlreef/frontend/-/merge_requests/407#note_325036571):  (+2 comments)

    > Looks really good! One little thing i noticed, the clickability of the left menu could be improved a bit. I think the best way would be to change the clickable area to the entire highlighted square. 
    > 
    > Now, folders can only be clicked in the "down arrow"; docs can only be clicked on the letters.

Story points: 1.0
Add visual hint for sequential nature of data processing pipelines

### Problem to Solve

We need to show the user that data processing pipelines are sequential, meaning that one operations follows the other. Currently, all three pipelines look the same, dispite the fact that data visualizations and experiment pipelines are not sequential but parrallel (the operations are built not on each other and are independent). 

This issue gives the user a hint that the operations in data processing pipelines are sequential. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/566ad5dc-edcf-49e3-7230-c4d0e1fa4078-205b/

![image](/uploads/b769a5e1f63ba034f2d991aab9824fb8/image.png)

Story points: 7.0
Enable production build on frontend

### Problem to Solve
In order to make the frontend deployment simpler and more reliable, we want to build the frontend in production mode. 

Think about serving the frontend via NGINX

### Technical Solution
Possibly use our NGINX as the webserver serving the frontend application

* [ ]  Use actual react production build `npm run build` instead of _develop server_ `npm start`
* [x]  Remove the SKIP_PREFLIGHT_CHECKS parameter to 
* [ ]  Add at least one integration test


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
* https://www.youtube.com/watch?v=wkJniy5FIbk
* https://codeburst.io/how-to-setup-nginx-for-react-a504f38f95ed

Story points: 9.0
Research payment gateway

### Problem to Solve
As a user I want to be able to choose between most convinient payment options (VISA, Paypal, etc?)

From business perspective, search the most covered, lowest payment commission. 

Option A: https://stripe.com/ 
*  Low payment fees (no montly, 1.4% Europe & 2.9% non-Europe + 0.25€ per transaction), many payment types including local (e.g. sofort), invoicing, recurring payments, dashboard, fraud detection
* [x]  Check on integration: https://stripe.com/docs/api

Option B: https://www.authorize.net/
*  Higher payment fees (25$ per month, 1.4% Europe & 2.9% non Europe + 0.30$ per transaction), many payment types not including locals, invoicing, recurring payment, dashboard, fraud detection.
* [x]  Check on integration: https://developer.authorize.net/

Both, Stripe and Authorize.net have
* a Java SDK
* a JavsScript SDK
* a quite extensive Developer documentation
* support for recurring billing and trial periods

**Stripe** additionally has much more built-in support for subscriptions:
* a manager GUI for creating subscription payments https://stripe.com/en-at/billing
* full-stack (frontend and backend) examples of how to implement the payment process
* full-stack (frontend and backend) examples of how to implement a subscription process



Story points: 1.0
Replace "SamerSBN" with official versions

### Problem to Solve
In order to make deployment, development, and maintenance of all services easier, we want to  "SammerSBN's" docker images with official versions everywhere (frontend/docker-compose, backend/docker-compose, backend/CI tests)

### Technical Solution


### Terminology


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 4.0
Run pipeline with docker locally (export as .rar)

### Problem to Solve
As a User I want to be able to run my pipeline locally. This can be done by creating the pipeline in MLReef and downloading the composed docker as .rar. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 16.0