giseldo/neodataset huggingface dataset:
Experiments: Training info

### Problem to Solve
In the experiments executive summary view the Training tab shows on what parameters the experiment pipeline was run. The info can be extracted from the experiment/xxx yaml file. We currently have two classes of parameters

a) required or standard parameters: these are required for running the experiment and are explicitly tracked in the yaml file

b) advanced parameters: these have preset values which are defined in the data datatypes.js (location FE web?). If the values have been changed while running a parameter, these values need to be overwritten by the values in the experiment/xxx yaml file. 

> For now we simply can list all parameters (standard and advanced) each in one paragraph


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

![image](/uploads/3da9c9ebdf27cce0736309561b8e0dfa/image.png)

Story points: 6.0
Insights / Show jobs list

### Problem to Solve
As a user I want to see all past and current jobs. A job is for now a pipeline that hast started. 



### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology


Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/07843c33-3b31-43b5-6d0f-10684160726c-71b5/

![Insights_Jobs_B](/uploads/c4ca7499aa28fab05221af7dce02bb50/Insights_Jobs_B.png)

![Insights_Jobs_A](/uploads/37662b248f8cbed035a7d1c2b497b816/Insights_Jobs_A.png)

![Insights_Jobs_C](/uploads/eb3fc7964eefc8d092292152a9ff1f62/Insights_Jobs_C.png)

Story points: 4.0
Adapt pipeline modal (loading view)

### Problem to Solve
The current **last modal view** for creating data processing, data visualization and experiments is almost entirely static. 

Until further technical solutions are created, we will give a more lively feedback by integrating a indetermined progress bar and links to the new data isntances, data visualizations and experiments.

* [ ]  Adapt experiment modal to show experiment created with key infos and dismiss others, (e.g.hyperopt) (@cpmlreef adapt view)
* [ ]  Adapt view for visualization + processing **singular only pipelines**

### Technical Solution

For the progress bar, use an inderterminate progress bar: https://material-ui.com/components/progress/

Backend won't be ready for sending the percentage so, the indeterminate is the one viable solution for the moment.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD for **data processing** modal: https://xd.adobe.com/spec/c0c2c46c-388c-4c39-518b-3af4fb97ecc9-b6f3/

Link to XD for **data visualization** modal: https://xd.adobe.com/view/528d7eb2-3422-4fc3-6a33-8c4ba4aaa4cb-24bc/

Link to XD for **experiment** modal: https://xd.adobe.com/spec/43c770a4-40c4-4b40-5917-6849c1913037-35f6/

![Processing_Data_C](/uploads/f22980632b95bd83fd9520f45f54834a/Processing_Data_C.png)

Story points: 4.0
Limit commit message in commit overview

### Observed behaviour
The commit message is to long and therefore, distorts the representation of commits in the commits overview (see additional notes).

### Expected behaviour
We need to limit the commit message to 40 characters in order to avoid this error. 

### Steps to reproduce
Go to Sign language classifier, then to commits in the blue nav bar.


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test
* A viable solution would be to add a collapsible text so that user can see all the commit msg



Additional Notes
=====================
![image](/uploads/b62b53444e58aff9a4c94f45a4baa92b/image.png)

relates to:

Story points: 2.0
Move delete project function to settings/general

### Problem to Solve
As a user (admin/owner of a repository) I want to be able to delete a project. Currently, this function is present but in the "wrong" section (in project overview page). 

This page will change completely with the new marketplace #183 

A new view will be created by @cpmlreef for this feature.


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 4.0
Integrate navbar counters

### Problem to Solve
As a user I want to be able to see the number of branches, commits, visualizations and so on in the data repository overview - blue navbar. 

These are the counters (marked with x are done, the others we would need to create):

* [x]  Commits
* [x]  Branches
* [x]  Visualizations
* [x]  Data Instances
* [ ]  Contributors


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


Additional Notes
=====================
includes / duplicates #138 

Project stats: ![image](/uploads/658e6294e83369c58543dc4812dcaada/image.png)


Screenshot: 

![image](/uploads/d275ec41e4139baa3a7211afa05f3638/image.png)

Story points: 4.0
Select + deselect function in data select modal (all pipelines)

### Problem to Solve
As a user I want to be able to easily select or deselect possible files in the data select modal view in pipelines: 

"Select" should only select all files and folders within the current folder.

Meaning, if the user navigates inside the `data/` folder and then clicks *Select all*, all the files and folders within the `data/` are selected but none outside.


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
![image](/uploads/7284263b84ba4df68d46169bebdad619/image.png)

Story points: 2.0
Display images in Readme viewer

### Problem to Solve
As a user I want to be able to see images in the readme. This is currently not possible as "quote from @SaathvikT": 
*This was an issue I had at the beginning as well because gitlab displays images in readme by parsing them in a different way i.e. it throws them as "img" tags to the frontend but these "img" tags are converted to text (so they are not really images, if it makes sense).*


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 10.0
Http 500 Errors in Data Populator

### Observed behaviour
Gitlab throws an HTTP 500 error on the first API call after installation


### Expected behaviour
Gitlab answers with 201 Created


### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 2.0
Backend: Setup integrationTests for whole stack including gitlab

### Problem to Solve


### Technical Solution


* [x]  Add integrationTest gradle task
* [x]  Add at least one integration test
* [ ]  ~~Replace Jedis with lettuce due to version conflicts~~
* [x]  Setup postgres, redis as easy TestContainers
* [x]  Setup postgres, redis, gitlab (via testcontainers docker-compose)
* [x]  Run real integration tests against gitlab

### Terminology


Additional Notes
=====================

* https://www.testcontainers.org/
* https://www.testcontainers.org/modules/docker_compose/

Story points: 4.0
Create user documentation (Alpha 1)

### Problem to Solve
As a *Alpha 1* user I want to have a basic documentation about how MLReef works. The user documentation should be accessible through our website and be based on markdown (hosted in the frontend repository) including a table of content. 


### Technical Solution
These are the steps to create the documentation

* [x]  Create concept and first layout
* [x]  Create main page
* [x]  Create subpage structure
* [x]  Create instructions for main pipelines and functions
* [ ]  Incorporate md to new website

### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 3.0
Data visualization - Overview

### Problem to Solve
The output of a data visualization pipeline needs to be stored in a seperate view (similar to data instance view). 

![Data_Instances_Overview_A](/uploads/733feddb0d545bffc50c0de28e6a0bbe/Data_Instances_Overview_A.png)

Link to XD: https://xd.adobe.com/spec/b52ce94c-57ce-4470-5b78-d2188589dd5a-c31d/


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 2.0
CSS: Wrong selection button for data types

### Observed behaviour
The button for selection the data type is only clickable at the checkmark, but not the whole button and the style is not 100% right to the specification.
![image](/uploads/0856494a020788c4cc62c4f16d22ad67/image.png)


### Expected behaviour
The button should be completely clickable and be of this style: 

![image](/uploads/5333421fbe74e764f728d7714ea29f7d/image.png)

See link to XD file: https://xd.adobe.com/view/c1bef4a8-42a9-40b0-6669-00305c4adbd7-b78f/


### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

Link to XD file: https://xd.adobe.com/view/c1bef4a8-42a9-40b0-6669-00305c4adbd7-b78f/

Story points: 2.0
Rest-Endpoint: CRUD DataProcessors

### Problem to Solve

Add DataProcessors CRUD and connect with CodeProjects
Can be created with REST and through python parsing.

Unique in "slug" and "branch", must be versioned/scoped as data-op-slug:branchname

### Technical Solution

* [x]  CREATE DataProcessors via REST
* [x]  Retrieve all DataProcessors via Rest: limit to VISIBILITY public and viewable by

Story points: 10.0
Rest-Endpoint: CRUD DataProjects

### Problem to Solve

As a User I want to retrieve all my DataProjects via MLReef.
Therefore, the REST has to provide basic CRUD for DataProjects

* Retrieve all my (viewable) DataProjects
* Create, with gitlab project creation and EPF-Bot
* Delete and Update, with gitlab creation

### Technical Solution

* [x]  Add endpoints
* [x]  Add gitlab connection for create and delete
* [x]  Add rest docs

Story points: 6.0
EPF Can load Data Operation and Algorithms from MLReef repositories

### Problem to Solve
As a User, I want to be able to run community and custom data operations in my pipelines. In order to do that the EPF has to be able to execute Data Operations and Model Algorithms from any public, team, and private repository hosted on MLReef.

These operations must either be actively included in the EPF during the publishing process, or downloaded and executed by the EPF on the fly, or of course a combination of the two. 


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.

* [x]  R&D if pip is useful or an overkill
* [ ]  assert that git clone works for public repos
* [x]  ~~R&D: add concept how backend sets permissions of EPF-Bot to allow git clone~~
* [x]  make EPF use a token for a EPF-Bot user
* [x]  ~~create an EPF-Bot user for every DataProject~~
* [x]  give EPF-Bot the needed permissions on used DataProcessors
* [ ]  add docu

Story points: 40.0
Find testing data set

### Problem to Solve
We want to test GIT / GITLFS on local gitlab instance. For this, we need to find a good data set at least above 20GB. 
Ideally, this data set can be kept for Alpha 1 usage.


### Technical Solution


Additional Notes
=====================
Link: https://www.kaggle.com/romainpessia/artificial-lunar-rocky-landscape-dataset

Link: https://www.kaggle.com/paultimothymooney/pain-pills-in-the-usa

Link: https://www.kaggle.com/diyer22/retail-product-checkout-dataset#20180824-13-36-25-2.jpg

CERN: http://opendata.cern.ch/record/43

Story points: 1.0
Kubernetes: Research and concept with Runners and GPUs

### Problem to Solve

Evaluate and research topics around kubernetes:

* [x]  Check if Gitlab CI runners can be spawned via Kubernetes
* [x]  Check if Kubernetes and GPU machines work together
* [x]  does Kubernetes have a concept of "nodes", e.g. "run DB on node A and web on node B"?
* [x]  does Kubernetes have a concept of node types, e.g. "GPU node" or "fast node"?


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Provide some research docu
* [ ]  If useful: Add documentation to the readme

Story points: 4.0
ReverseProxy: Solve CORS, Routing and SSL violations

### Problem to Solve
Our three services run at different ports, and thus technically on different hosts.
* frontend port 80
* Gitlab port 10080
* backend port 20080

Modern browsers have protection against Cross-origin resource sharing across different hosts. We have also seen some SSL problems with this setup.

This has to be solved.


### Technical Solution

Probably a Reverse proxy will solve any CORS issues we are currently facing


Additional Notes
=====================
 * Pretty supported project: https://github.com/jwilder/nginx-proxy
 * https://www.freecodecamp.org/news/docker-nginx-letsencrypt-easy-secure-reverse-proxy-40165ba3aee2/
 * https://serverfault.com/questions/229945/what-are-the-differences-between-haproxy-and-ngnix-in-reverse-proxy-mode
 * ngnix
 * haproxy
 * Quarcus
 * https://security.stackexchange.com/questions/48347/documented-best-practices-for-reverse-proxy-implementation

Story points: 4.0
Adapt Market Place View to new style

### Technical solution

Adapt the current "project-overview" page to the new look: 

![Market_Place_alpha1](/uploads/bd943ba0f1b94a1cd39bb539a58d2b1b/Market_Place_alpha1.png)

Link to XD: https://xd.adobe.com/view/5d9dcfd2-3bfb-4ee0-5069-8c9e2f0a5b7b-2bac/

Please use only the second artboard (no highlighted projects). 

### Questions

1. As you can see, there is a header nav-bar. Currenlty, we only have the ML projects, but hopefully soon we will be able to have the others as well. Does it make sense to code this navbar now, as we only have data repositories for Alpha 1?

Story points: 10.0
Failing test for Fileview

### Problem to solve

The tests are currently not working for the fileView set, the error is because of the configuration of the test which is not receiving a store object.

### Technical solution
Utilize the newly added store factory to retrieve a store object, which provides an initial state and lets the test run.

![image](/uploads/ed6fff5bb9e52e7fc60ae4e9d19a4990/image.png)

Story points: 2.0
Commits view error & weird behaviour

### Problem to solve
![image](/uploads/8529c6a8666e2103e869b004452e3fad/image.png)
![image](/uploads/2f6505a8cb0c5fc04dd6844dd1aab943/image.png)
When branches dropdown is clicked in the commits view, link changes weirdly, it should be reviewed and corrected.
The exception is raised when an option is clicked. 

### Technical solution
Control the exception by checking the nullity of the prop or look for the root of the error.


#### notes:
Wait until MR for #147 is merged for avoiding conflicts

Story points: 2.0
Load previous Data Operation configuration

### Problem to Solve
As a Data Scientist, I want to be able to load the data operation configuration from a previous experiment.


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme or the code


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

relates to: #150   

Story points: 1.0
Artifact download from experiment pipelines

relates to: [epf#48](https://gitlab.com/mlreef/epf/issues/48)

blocked by: #188 


### Feature
As a user I want to access the output files of a trained model. These are noremally the model itself and other relevant files, such as CSV files measuring the training performance. 

As a user i want to discover the files and also be able to delete the files (in case i do need to free storage space).

The user can access the files via the FE through the experiment details view: 


### Technical solution

The API provided by gitlab **should** be enough for implementing this functionality.


### Open questions

![Experiment_Details_Files](/uploads/204dd81d1602e396e48bc330cda82bf7/Experiment_Details_Files.png)

Link to XD: https://xd.adobe.com/view/acf7fd8b-01b4-4903-65a6-c348a4417cb6-dac6/


- We still need to define, if deleting the files in this view is a good option. Further, maybe it makes more sense to have a delete button somewhere else to delete the entire experiment (including all files). 

Story points: 1.0
Delete a single File

### Feature

The delete function is accessed in the file view directly, as shown here: 

![image](/uploads/fb5d91c9f6382daa656b39ca483614fe/image.png)

Clicking on delete will open this modal view (very similar on how GitLab does it): 

![Delete_File](/uploads/9f49da32f6b94f41c1b3a23e6516bb4e/Delete_File.png)

When the user writes a different target branch (existing or completely new one) the selector "start new MR... appears":

![Delete_File_A](/uploads/402ea7cee0253d9386f02dbb471123c1/Delete_File_A.png)

Link to XD file: https://xd.adobe.com/spec/d4e2618b-1cab-4d2d-7224-75835533f6d2-a2d3/

### Technical solution

All variants have to be included in this ticket: 

- [x] target branch is the same = simple commit 
- [x] new target branch = creates new branch
- [x] new target branch + start new MR selected = creates new branch + starts new MR

Story points: 1.0
Show Merge Request Bar when appropriate

### Feature 

This additional information bar shows the user what commit status his current branch has: 

![image](/uploads/b0ab7d90ce8d1a75c4f0ed92e38a5584/image.png)

This is very similar to how GitLab does it in the Branch menu: 

![image](/uploads/04e1a7c07e12a84685387d35122bbda0/image.png)

This bar does never show up when user is in "Master".

Story points: 2.0
Create New ML Project

### Problem to solve
We need to be able to create a new ML project (repository). 

This form is very similar to how GitLab does it. We, in addition, add the "choose data type" function, as this will define, in what category the project can be found in the market place. 

As of now I think the best way to do this is by adding tags to the project: e.g. image, video as tags. 


### Technical solution

* [ ]  Update the [handbook page](https://gitlab.com/mlreef/www-mlreef-com/blob/master/handbook/engineering/sops/create-gitlab-project.md) on creating projects (delete the Machine Learning projects section)


Additional Notes
===================

![New_project](/uploads/035d6f663da385a314e3edc3b71dacb3/New_project.png)

Link to XD: https://xd.adobe.com/view/c1bef4a8-42a9-40b0-6669-00305c4adbd7-b78f/


Story points: 1.0
Data Visualization "Drag & Drop" Field

### Problem to solve

Currently, the data visualization pipeline does NOT allow any pipeline creation, as the data visualizations can NOT be drag&droped to the center field. 

![image](/uploads/cfb067a392b778a8ae766b12ba6923af/image.png)


### Technical Solution
Use the same functionality provided in the data pipeline and experiment pipeline views. 

![image](/uploads/45ae88f09dfafe4d7dc0def06f6c7a8a/image.png)

Story points: 1.0
Show experiment information in experiments executive summary view

### Problem to Solve

One of the most important parts in MLReef is experiment reproducibility. This is shown in this view: 

![image](/uploads/f67bf553f771df7fd645545463c31917/image.png)

For Alpha 1, we need to be able to show the information for: 

- [ ] Data
- [ ] Training

The information in the middle Tab "Algorithm" will only display the name of the algorithm used (Title). 

The information that will be displayed is, in the first iteration, NOT clickable (meaning it does not provide further information, e.g. when clicking on "123 data files" the user does not get the data files shown. This will be solved in subsequent tickets.

This ticket needs to be coordinated with the development of our EPF (@flohin, @vaibhavmehrotra). 

Please also provide instructions (@all) if this ticket should be subdivided in smaller tickets.

### Technical Solution

Please provide here the technical solution for this ticket. Especially, how is the information stored and how can the FE read it.

Story points: 5.0
Validate pipeline input earlier

### Problem to solve

Currently, the FE tests the input variables in a pipeline in the modal view after pressing "execute": 

![image](/uploads/1714c59b014c20e3cf0b25648bbb88e1/image.png)

The test should start one step before, as there the feedback of incorrect input variables is best: 

![image](/uploads/667eada8118c00a4d74d9673014e4962/image.png)

Story points: 1.0