giseldo/neodataset huggingface dataset:
Load Pipeline Configuration

### Problem to Solve
As a Data Scientist, I want to be able to load the experiment configuration from a previous experiment.

An executed data pipeline will create a config file describing what exact components it used (the `mlreef.yml`)

To recreate a data pipeline a user shall simply click the "view pipeline" button in the data instance view and be directed to an automatically pre-filled data pipeline view like this one, where all parameters and operations are selected according to the saved config file: 


### Technical Implementation

* [x]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme or the code

* There are two alternatives for solving this issue:
1. Parse the .mlreef.yml file to get the operations, folder selected and parameters entered by the user.
2. Persist pipelines related information in a database(maybe a snippet), so that the pipeline view can be rebuilt from that source.

Additional Notes
====================
![image](/uploads/9f0f822dfed388a010b52d36d7767deb/image.png)

![image](/uploads/a681cfa2d9855b981041da3b9cd293c6/image.png)

relates to: #164 

Story points: 5.0
Combine same experiments cards

### Problem to Solve
This ticket is very similar to ticket #119 - Combine same cards.

Same experiment cards in the data instance overview should be grouped together, looking like this (for expired cards - same for any other label): 

![image](/uploads/b5f0d04cadbbbcab2b3a64be99c4fc92/image.png)

Currently same experiment cards are not grouped and it looks like this: 

![image](/uploads/8887b092affb8046f9672b80facdfa81/image.png)

### Technical Solution
Similar solution as in commits overview - here same aged (in days) commits are grouped togther.

Story points: 6.0
Commit Details with Diff view on Images

### Feature

A commit with images can have either/and these scenarios: 
1. Add an image 
2. Delete an image
3. Change an existing image (Diff view!) 

This commit detail view should look like this, dependent on what scenario the commit has: 

![Commits_Details_A](/uploads/8b3514a76cdd53f34f13d93573b9a2fa/Commits_Details_A.png)

Link to XD file: https://xd.adobe.com/spec/98efca21-bcd1-40e6-5430-8855fd3efe7d-eede/

Story points: 5.0
Limit Commit message to GUI size

### Problem to solve

Currently, the last commit message is to long, which takes much of the available space and squeezes the profile avatar as shown here: 

![image](/uploads/8e6bc8c463c8801b393900b6eab5b9f6/image.png)

### Technical solution

Limit to a maximum of 40 characters the commit message.

Story points: 1.0
Create new Branch

Simple form to create of a new branch from an existing branch. 

Link to XD file: https://xd.adobe.com/spec/254df67a-f28e-4fff-5703-2a4183dde1e6-537c/

![Data_Rep_Overview_A___1](/uploads/1d775b9ee93647257390fdee96ad444a/Data_Rep_Overview_A___1.png)

![Data_Rep_Overview_A](/uploads/4fd340b8a94ab2bda9d85df7ce4fa732/Data_Rep_Overview_A.png)

Story points: 1.0
Login in Frontend via Gitlab/Backend: Research and concept

### Problem to Solve

Users need to login via their own gitlab account.
Login mask consists of username/password, which should somehow lead to a valid authentification with gitlab.
Gitlab is now our "user store", therefore, direct authentication is encouraged.

### Technical Solution

* [x]  Research, if we can forward credentials to gitlab and retrieve the token
* [x]  Research only, if we can create a token via gitlab api


See the following tickets:

* Backend-Issue: https://gitlab.com/mlreef/backend/issues/18
* Backend-MR: https://gitlab.com/mlreef/backend/merge_requests/16
* Fronted-Issue: https://gitlab.com/mlreef/frontend/issues/167

Story points: 5.0
Forked Project does not appear in the My Projects List

### Observed behaviour
After forking a Project, the new project does not appear immediately in the "my-projects" overview


### Expected behaviour
After clicking the fork button the new project should appear in the my projects list


### Steps to reproduce
* Login to master
* Go to explore projects
* Click on either Ham10000, SAR-ESA, or Demo
* Click Fork
* Go back to the my-projects view


### Solution
* [ ]  Add at least one unit test
* [x]  Add documentation to the readme or the code where it makes sense



Additional Notes
=====================


relates to:

Story points: 1.0
Data Visualization - Pipeline

### Problem to solve

Create the execution process and modal view for executing data visualization pipelines. 

![Data_Visualization_A](/uploads/bcb87b276d97bd82d2e6c761c0d24d3e/Data_Visualization_A.png)

![Data_visualization_C](/uploads/0da9032a7d218a6132cc6328affd3c2a/Data_visualization_C.png)

![Data_visualization_B](/uploads/257d143b17b626e21045ff0708cfc99d/Data_visualization_B.png)

Link to XD: https://xd.adobe.com/view/528d7eb2-3422-4fc3-6a33-8c4ba4aaa4cb-24bc/

### Technical solution

Reuse the processes and modal view from another pipeline (e.g. data processing pipeline). The output of a data visualization is stored as a visualization/pipeline_name branch. 

In the visualization overview this complete branch name can be used to filter out only visualizations (similar how it is done with data instances). 



Story points: 1.0
Add Loading for the Data Repo File Tree

### Problem to solve

The loading of the different content in the data repo overview takes to long. The current version, the loading spinner, is to intrusive and blocks everything. 

We need asynchonous loading in this order:
1. Loading of the folder tree and files 
2. Loading of "last commit and last date"
3. Loading of the weights of files and quantity of files in a folder


### Technical Implementation



## Additional Notes
While the data for "last commit", and "last date" is beeing loaded it can just be left blank and appear as soon as the data is available. As seen here:

![image](/uploads/28265ec2d2b118faeb83ba54b8ad4901/image.png)

To show the user that something is going on, we replace the MLReef logo in the navbar with this GIF (so does it GitLab as well): 

![MLReef_w](/uploads/2330ef8acba3e6fc8e45c71488528140/MLReef_w.gif)

![image](/uploads/4a20611b379f1f9b2332cd4537bd79f4/image.png)

Additional notes:

Files API has a limit of 100 files, therefore to calculate a folder size cannot be done via API requests. For future tickets think about weight and count files in the backend and persist that information, so that there will be always a kind of report to ease the access and calculations.

To get the last commit information, it is possible to try in the future with its own gitlab instance the next API:
https://gitlab.com/mlreef/frontend/refs/develop/logs_tree/.vscode?format=json

For the moment, we cannot fully access this API but our own instance can be a solution.

Story points: 2.0
Data Repo Overview Menus and Functions

the repository overview has a number of different drop down menus and functions.
We should include the components of the variants as described in the XD files.

Link to original data repo XD files: https://xd.adobe.com/spec/6af7ea58-6fe4-4a18-5db1-9c5e7dd598f2-1cc6/grid

In variant B: No deeper functionallity here.
In variant C: The search function will not work, but showing the branches should (with working links). Also the file view for images needs to work!
In variant D: Dropdown should work, but no deeper functionality needed for the moment. Clone no need to work.

Story points: 5.0
Ensure Model Output, Artifact Upload, and Frontend Artifact Download

### Problem to Solve
When a model is trained successfully it outputs a binary file. This file **should** be placed at the location `output-path`. The binary should be uploaded as a Gitlab Artifact by properly configuring the `mlreef.yml`.

Later, a User shall be able to download the Artifact from the Frontend. 

The Scope of this ticket is to make the download work end2end


### Technical Solution
- [x] fix -> https://gitlab.com/mlreef/mlreef/-/merge_requests/550#note_371628096

### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Story points: 13.0
Update to Node 14

Our current Node Version Node 10, even tough an LTS version is nearing its end of life:

![image](/uploads/64259d503e24fabe7d43388461973c2b/image.png)

In order to stay up to date with security updates and other library updates we should update to the next LTS version of node … which is Node 14.

For this at least the following packages have to be updated:
* SCSS to 4.14+ [more info …](https://github.com/sass/node-sass#node-sass)
* Dockerfile: `web/Dockerfile`
* Script: `web/bin/node`

Story points: 10.0
revise design on registration screens

### Observed behaviour

The design seems "out of date" for the registration screens and need revision. 

* [ ] The dropdown menu after registration needs more entries (currently only data scientist and student): 
* Data scientist
* Developer
* ML engineer
* Researcher
* Student
* Team lead

* [ ] Toastr message on sucessfull registration: Your registration was successful. Welcome to MLReef!

### Expected behaviour



### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

See XD file for design: https://xd.adobe.com/view/c4069ef1-b6d4-44f5-528c-4e0da5800e5a-b237/

relates to:

Story points: 5.0
Error on registration

### Observed behaviour

The following error is given when trying to register: 

![image](/uploads/f8d994fa03bd6922a47318a2d032009f/image.png)

### Expected behaviour



### Steps to reproduce

Open up master branch (through mlreef.com), go sign in, go create an account and then fill all fields in. 

### Terminology
> Define terms that are important in the context of this task



### Solution

@diegovinie lets discuss when and how we want to follow up on this bug and coordinate to open up MLReef for the general public. 



Additional Notes
=====================


relates to:

Story points: 1.0
Ensure Reserved Names are Denied

### Problem to Solve
We need to save some namespaces in projects/users/groups for operational tasks, such as (need to write this in the docs as well!): 

Not all project & group names are allowed because they would conflict with existing routes used by GitLab.

Scope of this ticket includes:
* [ ] Unit Tests for all disallowed words (TDD)
* [x] Documentation in MLReef's REST Docs
* [ ] Documentation in MLReef's User Docs
* [ ] Collaboration with the Frontend team for adaptation of the FE pages Create Project, Create User, and Create Group for using the new endpoint


### Technical Solution

Suggestion: In order to centralise valid names one (or multiple) backend endpoints should be created which allow the frontend to query if a certain string is a valid name.

The response shall contain 
* TRUE/FALSE 
* The `slug` of the queried name
* ErrorMsg: If the name has already been taken by another user, group, project, etc...)


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
**Reserved project and group names**

For a list of words that are not allowed to be used as group or project names, see the [`path_regex.rb` file](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/path_regex.rb) under the `TOP_LEVEL_ROUTES`, `PROJECT_WILDCARD_ROUTES` and `GROUP_ROUTES` lists:

* `TOP_LEVEL_ROUTES`: are names that are reserved as usernames or top level groups
* `PROJECT_WILDCARD_ROUTES`: are names that are reserved for child groups or projects.
* `GROUP_ROUTES`: are names that are reserved for all groups or projects.


**Reserved project names**

It is currently not possible to create a project with the following names:

* `\-`
* `badges`
* `blame`
* `blob`
* `builds`
* `commits`
* `create`
* `create_dir`
* `edit`
* `environments/folders`
* `files`
* `find_file`
* `gitlab-lfs/objects`
* `info/lfs/objects`
* `new`
* `preview`
* `raw`
* `refs`
* `tree`
* `update`
* `wikis`

**Reserved group names**

Currently the following names are reserved as top level groups:

* `\-`
* `.well-known`
* `404.html`
* `422.html`
* `500.html`
* `502.html`
* `503.html`
* `abuse_reports`
* `admin`
* `api`
* `apple-touch-icon-precomposed.png`
* `apple-touch-icon.png`
* `assets`
* `autocomplete`
* `dashboard`
* `deploy.html`
* `explore`
* `favicon.ico`
* `favicon.png`
* `files`
* `groups`
* `health_check`
* `help`
* `import`
* `invites`
* `jwt`
* `login`
* `oauth`
* `profile`
* `projects`
* `public`
* `robots.txt`
* `s`
* `search`
* `sent_notifications`
* `slash-command-logo.png`
* `snippets`
* `unsubscribes`
* `uploads`
* `users`
* `v2`

These group names are unavailable as subgroup names:

* `\-`

Story points: 3.0
Create model training example (tutorial)

### Problem to Solve

We want to create several tutorials as ML projects in MLReef to show our users how they can use MLReef and also learn. 

There is already one example of a tutoria: http://mlreef.com/mlreef/tutorial-cats-and-dogs/blob/master/README.md

Create:
- HTTPS: `https://gitlab.com/gitlab-org/gitlab.git`
<https://mlreef.com/mlreef/training-model.git>. 

### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 5.0
User Avatar is not displayed in project view

### Observed behaviour
User avatar of the latest commit section in the project view is not rendered.
![image](/uploads/7eef92beddad05addb22f1fc689820fa/image.png)


### Expected behaviour
Avatar is rendered in project view

### Steps to reproduce
1. Login.
2. Select the SLC(Sign Language Classifier) project.

Story points: 6.0
Create market place view for visitors

### Problem to Solve
Adapted view of the market place for a visitor (not authorized). 

### Technical Solution

Reference to: https://gitlab.com/mlreef/frontend/-/issues/539 for the last screen (loading screen).


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
Link to XD: https://xd.adobe.com/view/5eb0f210-a2c7-435c-4e25-7e43dc0887c4-2881/

Story points: 8.0
Selecting file and then selecting file again overrides the initial file

### Observed behaviour

When using the uploading file function in the data / code repository overview, the selected files to upload get overwritten as soon as I open the explorer popup when uploading a new file. 

### Expected behaviour

When selecting another file to upload (by exiting the explorer view and entering the explorer view again and selecting another file for upload) the second file should persist with the other file. 

This ticket may relate to these: 
* https://gitlab.com/mlreef/frontend/-/issues/534
* https://gitlab.com/mlreef/frontend/-/merge_requests/486


### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 2.0
Pre-uploading of files in "files upload"

### Problem to Solve

When I select files for upload, as soon as they appear in the list, they should automatically be uploaded. When doing so the blue bar on top of each file should fill up as the files are uploaded. 

At the moment, the files are just listed with the blue bar at 100% from the very beginning. 


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

![image](/uploads/27a83be8e9169fc50c14c42cb03eb742/image.png)

Story points: 4.0
Backend: Improve Experiment's lifecycle

### Problem to Solve

Right now the Experiments have an UUID and a nullable PipelineInfo.
The frontend wants to use a "number", this exists currently within the PipelineInfo.

IF an Experiment gets created *and* started, an Experiment has a PipelineInfo with that number (locally unique per DataProject), but if it is just created, no such PipelineInfo and therefore number exist.

This could be strange in some situations, so it is still / again technical debt:

### Technical Solution
* Add another number, which is scoped in MLReef, does not depend on gitlab, and exists at the exactly same time, the experiment gets created
* add a constraint: experiment_number, data_project_id
* (and also for PipelineCOnfigs): pipeline_number, data_project_id
* add semaphor/synchronized, concurrency control to avoid dunplicate local ids

Story points: 3.0
Merging does not respond

### Observed behaviour

When trying to merge an MR in the MR detail view, the merge doesnt respond. 

### Expected behaviour



### Steps to reproduce

* In develop, in any existing or new project
* create a branch 
* add a file
* Create a MR and try to merge the changes.
* I used one cat image as test.

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/d8419fd7bf4e9dc5a02e024020831c4b/image.png)

Story points: 8.0
Backend: Reset Password Process

### Problem to Solve
In order to allow users to reset their passwords, we have to implement a reset-password process in the backend.

* Upon triggering the process, the User shall receive an email which contains a password reset link which contains some form of a one-time password reset token.
  * the token can only be used once
  * the token shall expire after 24 hours
  * the token shall expire immediately if a new token is issued
* After following this link they can enter their new password in the frontend.
* The frontend then sends the password, together with the token to the backend.
* The backend then:
  * validates the token 
  * saves the new password
  * expires the token
  * performs a login for the user
  * returns the Session token to the frontend, so the frontend can "also login" the user


Part of this ticket is also documenting and communicating this process sufficiently with the frontend developer who implements #500 

### Technical Solution
The current design requires two backend endpoints.
1. For the frontend to trigger the process
2. For the frontend to set the new password


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 8.0
Backend: Support Data Pipeline Creation

### Problem to Solve
In order to take full control over the execution and management of Data Pipelines, the backend has to offer an API endpoint for pipeline creation.

This ticket is to ensure that all necessary functionality, can be finished.


### Technical Solution
* see what is not working and hotfix stuff
* #459 : If name generation is removed there, we have to include it in this issue!
* attention: this issues introduces technical debt, but this is not fixable now. See #561 

### Actual Features
* [x] : Add name generation of #459 

Additional Notes
=====================

* https://mlreef.gitlab.io/backend/develop/#pipelines
* https://mlreef.gitlab.io/backend/develop/#pipelineinstances

Story points: 2.0
Website / Blog post

### Problem to Solve

This is the standard design and layout of our blog posts on our website. 

Blog image replaces the header section and has a paralax effect. The title and author have a dropshadow for better readibility.


### Technical Solution


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/19fe631f-d2dd-4b6e-5a1f-4f18ad8d790b-6fcf/

Story points: 1.0
Error creating data project - doesn't recognize initialize_with_readme

### Observed behaviour



### Expected behaviour
Create a project


### Steps to reproduce
Create a project with readme


### Terminology
https://mlreef.gitlab.io/backend/develop/#_post_data_projects



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
![image](/uploads/e7a49e59170b58ec5c67260b0290e166/image.png)


relates to:

Story points: 1.0
User_account / settings / profile

### Problem to Solve
As a User, I want to be able to manage my profile in MLReef. This menu is accessed in the user avatar dropdown under the settings.


### Technical Solution
Most, if not all, of the functionality covered in this ticket, is covered by the Gitlab API.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/811be39f-34d5-47ea-6bcc-95409cf5f74e-f72a/

![Settings_Profile](/uploads/2183264d484e50e92d43b876da3bcabc/Settings_Profile.png)

Story points: 14.0
Backend & EPF: Add static base environment

### Problem to Solve

For the publishing process we need to store base environments in the backend, to create some new docker containers on top of them.

Right now we think we need to store at least:
* docker_image: String "image:version"
* unique_slug: String "PYTHON_2-6-GPU"
* name: String "Python 2.6 Cuda GPU"
* description: String "This package is nice an nice python 2.6 installed" (can be used for description, requirements.txt, packages, and maybe python_version)
* requirements: String(Long) "represents file content of requirements.txt"
* machine_type: String(enum) CPU, GPU
* python_version : String "2.6" 


### Technical Solution


* [ ] add those fields and invent a Enum, this should be sufficient for now

Story points: 2.0