giseldo/neodataset huggingface dataset:
Deleting data sets does not work

### Observed behaviour

Deleting a data instance does not work (the button does not react)

### Expected behaviour

It should delete the data instance and the underlying data-pipeline branch. 

### Steps to reproduce

Go to the dev environment, 
* open cats and dogs project
* go to data sets 
* press the delete button to the modal view. 

### Terminology
> Define terms that are important in the context of this task



### Solution

* [ ]  Get the delete function to work
* [ ]  Adapt the view to the new design, link to XD: https://xd.adobe.com/spec/98f972a6-7f0d-4c5f-469a-65c191a35b3a-eec8/

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/c87efaaa8c94ea82a357789f6c895537/image.png)


relates to:

Story points: 4.0
Separate Data Pipeline input and output data

### Observed behaviour

When creating a pipeline (eg. data processing pipeline, data visualization or experiment), the resulting branch that is created contains from its start all files present in the origin branch. 


### Expected behaviour

We are currently using branches to store the output of a branch. As long as we have this system, the origin files should not be in the newly created pipeline branch (such as data-pipeline/XXX branch).


### Steps to reproduce

Open dev environment, start a processing pipeline, navigate to data instance (or in gitlab backend) to see the content of this branch.


### Terminology

@rainerkern do you think this ticket makes sense in the perspective of (hopefully) switching to a better storing mechanism (buckets)? 


### Solution
The technical implementation of this can look similar to how Maven does it. In Maven, there exists a general output folder and every script generates a separate folder within that. Subsequent scripts can then easily access the previous results.

We can (and probably should) do something similar. For example, if a pipeline consists of three scripts:

1. Every Data Pipeline script per default needs an input and an output folder.
2. The EPF (`mlreef.yaml`) takes the original `data` folder as input for the first script in the pipeline and directs the output to a temporary location (e.g. `output/1`)
3. The EPF uses `output/1` as input for the second script and directs its output to `output/2`
4. Then the EPF uses `output/2` as input for the third script and directs its output to `output/3`
5. Then, at last, the `data` folder is cleared and the contents from `output/3` are moved to `data` and subsequently persisted as the Data Instance

**Mixing Data:**
If the user decides that he wants to original input data and the output data mixed together, in step `5` the input data is **NOT** deleted from the `data` folder

This functionality can also be added for each step. So the data from each step is propagated to the output folder

@cpmlreef create designs in frontend.

**Storage:**
This description is still completely agnostic of the type of storage that will be used; be it S3, FTP, or GIT.

Story points: 6.0
Navigate in folder tree in select data modal

### Problem to Solve

As a User I want to be able to navigate in the folder structure to select files within folders. This is currently not possible, as only the very parent view of the data repository is shown (and not clickable = no navigation). 

### Technical Solution

The function should look the same as the folder tree navigation in the data repository view. 

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
Link to XD: https://xd.adobe.com/spec/88d8a453-2b64-43bf-71a6-a2a48c4ef2b7-fee9/

![image](/uploads/658fbb758ad1edf5dad206e69b9d92ea/image.png)

Story points: 4.0
Make modal views sexy from size

### Observed behaviour

Currently, nearly all modal views are not in a "propper" size. Either they are to big, not adapting to their content, to streched or to little. 

We need to make all of them sexy! ;)

### Expected behaviour

The modals need to be correct in size as of their design. 

### Steps to reproduce


### Terminology
> Define terms that are important in the context of this task



### Solution

This ticket relates very well to this one: frontend #251 

* [ ]  Add at least one test



Additional Notes
=====================

Modal in select data: 

![image](/uploads/d7afaecf7e302d4c8329a3c2bfdd22a1/image.png)

Execute pipeline modal: 
Either to streched 

![image](/uploads/931eb6335e2f9482f7fca0e4e3a30447/image.png)

or too little

![image](/uploads/c4699a492eefab6a33d9b5dd72f9ed6e/image.png)


relates to:

Story points: 4.0
File/Folder name in modal data select breakes to "early"

### Observed behaviour

The space for file and folder names in the modal view to select files / folders is to short, therefore the name breaks very "fast". 

### Expected behaviour



### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution

Change the space for names to it fits at least 30 characters for all modal views: 

* [ ]  Data visualization select data modal
* [ ]  Data processing pipeline select modal
* [ ]  Experiment pipeline select modal

Additional Notes
=====================

![image](/uploads/0bda59649abd80b0adfaf3817f9d6f40/image.png)

relates to:

Story points: 1.0
Deleted files have wrong lable in image diff view

### Observed behaviour

The diff view when files where deleted has currently the lable "source file" on it. This view exists for example in commit detailed views, when creating new Merge Requests or when reviewing MRs in the Changes tab. 

### Expected behaviour

It should state "deleted". 

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/7eceb98a58b06270075acd1df769932a/image.png)

relates to:

Story points: 1.0
Create merge request link from branch view does not work

### Observed behaviour

The link from the branch overview to create a new merge request from a branch does not work. 

### Expected behaviour

The button links to create new MR, where the information is already pre-filled (branch source name, diff views, commit history).

### Steps to reproduce

In develop env, retail project, branches.

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/70ac809aac673e1b5a2208b7b022573d/image.png)

relates to:

Story points: 1.0
Limit character count on file names to 40

### Problem to Solve
We have to limit the amount of characters displayed in the data repository to 40, as more will do breaks in line. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

![image](/uploads/321e2b1f59f054050a9b085d8686c2db/image.png)

Story points: 1.0
Link branch name to branch in new merge request view

### Problem to Solve
As a user i want to be able to review the branch by clicking on the branch name that I want to create a MR from. To do this, simply link the branch name to the corresponding branch in the data repository overview. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
By clicking on branch name
![image](/uploads/6326cc47759c2b44b43056d5606e1771/image.png)

it links to this: 

![image](/uploads/0fcccceeba7334e893dd60177c6dcd30/image.png)

Story points: 1.0
Forking a project repository doesnt seem to work

### Observed behaviour

When forking a project nothing happens. 

### Expected behaviour

After pressing the fork button in a "foreign" repository, the repo shows a message being forked and then the user lands on the forked repo overview page.

There are some logics that need to be considered when forking a project: 
* [x]  Forking a project is only possible, if the project is not the user´s own project. If it is the user´s own project, the project button is hidden away. 
* [ ]  Forking a foreign project puts the project into "my projects" tab
* [ ]  A project from the same group can be forked
* [ ]  A project from the same group will appear in the "my projects" tab

Relates to: frontend#357

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 4.0
Allow launching of multiple experiments in parallel


### Expected behaviour

For example drag and drop the same algorithm several times with different hyper-parameters set by the user.
The pipeline will run the experiments in parallel. The user can compare the results of all the experiments.

Other case can be drag and drop different algorithms with hyper-parameters set by the user.
The pipeline will run the experiments in parallel The user can compare which model is the best for the dataset.


### Steps to reproduce

Go into develop environment, open a new experiment pipeline, copy and adjust parameters for the models and execute the pipeline. To test open the experiment overview page or in our gitlab the repo CI/CD tab. 


### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 10.0
Pipeline warning: Fork project to gain access to functions

### Problem to Solve

There are some functions that need to be "dissabled" for projects that are not owned by the user. 

For example, a user will not be able to start an experiment within an open project by another user. 

If the user would want to do this, he first must fork the project and then create own experiments. The same accounts many different functions, such as creating new files, uploading files, etc. 

**This ticket needs design elements and detailed description on what functions are concerned!**

### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 10.0
Differenitate between data instance and normal branches in dropdown

### Problem to Solve

**Completed** data instances should show up in dropdown menus within the pipeline views as graphically separated from "normal" data repo branches. 

Currently, all data instances are filtered out in the dropdown views. 

Note: Only "normal" branches shall be shown in the data repo overview dropdown (as it is currently is handled)!

### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


Additional Notes
=====================

Link to XD: https://xd.adobe.com/spec/604591b4-1d03-4f45-4bf8-3da3b2759d8c-1674/

![Select_Data_Experiment_PPL_B](/uploads/022843b247ddba3a744ff07d99cb5162/Select_Data_Experiment_PPL_B.png)

Story points: 2.0
Setup PROD Deployment Pipeline

### Problem to Solve
Setup the deployment of a productive environment in the _Infrastructure_ repository.

Design a release pipeline which deploys **exactly** the same binaries to the prod environment that was deployed to the test environments earlier.

The Layout consists of three repositories: _Frontend_, _Backend_ and _Infrastructure_. Also in this setup, there is no need for an extra _develop_ branch in the Frontend repo.

* Development in bug/feature branch
  * Pipeline builds Frontend docker for branch
  * Pipeline deploys to dynamic _feature-branch-environment_ with new Frontend container and the Backend container referenced in the _docker-compose.yml_

so far nothing new here.

* Merge to master (get rid of the develop branch for now)
  * Pipeline deploys to static _develop-environment_ with the Backend container referenced in the _docker-compose.yml_
  * Triggers _Infrastructure_ pipeline 
  * Take _frontend:latest_ and _backend:latest_ and tag them _frontend:master-$pipline-id_ and _backend:master-$pipline-id_ respectively
  * Provide a manual job which runs the infrastructure pipeline to _frontend:master-$pipline-id_ and _backend:master-$pipline-id_ to production

* Infrastructure Pipeline
  * Provide a Job which deploys a specific set of docker images to PROD


**Additional:**
If we save the exact docker image tags which were used during a run for Frontend and Backend images, this would allow us to trigger the Infrastructure Pipeline from the Frontend Project as well as the backend Project and only change out the new parts; A Frontend trigger would update only the Frontend on the PROD environment without touching the backend.


### Technical Solution

* [ ]  If useful: Add documentation to the readme or the code


### Terminology
* **Instance/Environment**: A separate "server" running on EC2 hosting a complete, independent installation of all microservices and databases of MLReef 
* **DEV/develop Environment:** The Environment used by developers for development. Can break at any time.
* **TEST Environment:** A more stable environment used for User Acceptance Testing. Should be quite stable already. `DOES NOT CURRENTLY EXIST`
* **PROD/production:** The one and only real production environment
* **staging:** Not a real Environment but rather a temporary place to stage new versions for production deployment. `DOES NOT CURRENTLY EXIST`
* **PipelineId:** A globally unique integer, automatically generated by Gitlab, reasonably human-readable, and increases in number.

 
Additional Notes
-------------------
Download Artifacts from other (most recent) pipelines [Gitlab ticket](https://gitlab.com/gitlab-org/gitlab-runner/issues/2656)



Story points: 10.0
Better AMI (Amazon Machine Image) for Gitlab Runners

### Problem to Solve
Gitlab Runners can be deployed in two different modes. The single runner which just accepts a job and executes it directly. And the multi-runner; which uses a separate driver to create AWS EC2 instances and execute jobs on them. This would offer us much more parallel computing power.

The driver uses a configurable Amazon Machine Image (AMI) to create the new EC2 instance.

In the Frontend project, the multi runner configuration exists, but the single runner config is used. It is located inside deployment script script -> [src/bin/deploy.sh](https://gitlab.com/mlreef/frontend/-/blob/develop/src/bin/deploy.sh)

In order to be able to harness the power of GPUs in (future) elastically deployed Gitlab runners we need an AMI which already contains NVIDIA Cuda drivers.

### Technical Solution
**Preferably find** or otherwise create a more specialised AMI (Amazon Machine Image) which already has the following software preinstalled:
* Docker in the newest version
* Nvidia Cuda drivers

look to the [ec2 startup script](https://gitlab.com/mlreef/frontend/-/blob/develop/src/bin/startup.sh) for guidance

Additionally:
* [ ]  Add documentation to the readme or the code

Story points: 5.0
Make Develop instance more accessible for developers

### Problem to Solve
As a Developer, I like to use my locally running frontend while connected to the `develop` instance of the backend and Gitlab.

Currently, when doing so, CORS errors are thrown by the remote backend.

### Technical Solution
* [x]  Add documentation in the Frontend's getting started guide how to connect your local instance to the DEVELOP infrastructure
* [x]  Implement CORS headers in the Backend #280 


Additional Notes
=====================
Error, when trying to connect to the aws instance.
![image](/uploads/85cff25740227bf4a54406a64f14ff82/image.png)

Story points: 1.0
Integration and Proxy as MicroService-Holder

### Problem to Solve

In the end, we want to have a working infrastructure hosted on AWS with a certain URL.
Some ports are publicly available, others must be present for maintenance (ssh).

We have to assert the following is fixed:
* frontend is available as HTTP/HTTPS
* backend is available as HTTP/HTTPS
* gitlab is available as HTTP/HTTPS
* No cors violations are happening
* frontend can talk to backend & gitlab
* backend can talk to gitlab

I propose the following **Solution A**:
* have the reverse proxy as the single domain entry point **mlreef.com**
* proxies mlreef.com/api to backend:8080/api 
* proxies mlreef.com/gitlab to gitlab:80/gitlab with relativeRootUrl /gitlab
* configure gitlab to use the relativeRootUrl /gitlab to maintain proper redirects (mlreef.com/gitlab/user/signin, e.g.)
* frontend uses just /gitlab as the gitlab root URL, with the proxy active URLs get a proper rewrite. 
  * Attention: In my understanding, the API would be on /gitlab/api/v4 instead of OTHER_HOST:PORT/api/v4, be aware of that extra /gitlab in that URL
* frontend uses just /api as the service root URL, with the proxy active URLs get a proper rewrite. 
* All services can run with http. The reverse proxy can be configured to act as an https endpoint

### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [x] Discuss this and read into reverse-proxying
* [x] Set gitlab relativeRootUrl  in docker-compose
* [x] fix nginx config: add /gitlab to gitlab:/gitlab instead of /gitlab to gitlab:/
* [x] adapt prodution frontend to just use /gitlab and dont need any root url variables (Also: CORS!)
* [x] frontend should be able to choose different url when run in development

Additional Notes
=====================
* https://stackoverflow.com/questions/35756663/api-gateway-vs-reverse-proxy
* https://stackoverflow.com/questions/43548363/using-api-gateway-as-reverse-proxy-for-web-applications
* https://stackoverflow.com/questions/50333994/node-js-api-gateway-nginx-as-reverse-proxy

Story points: 4.0
Include status icons on running pipeline summary cards

### Problem to Solve
As a user I want to easily observe if a pipeline is still running. The easiest way is to include a spinner in the "running" pipeline cards (see XD link). 

This needs to be done for: 
* [ ]  Data visualizations
* [ ]  Data instances
* [ ]  Experiments


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

XD link: https://xd.adobe.com/view/209888ea-007c-4cd4-6cff-e246d446ebd9-56a9/

![image](/uploads/50bf9eea6060afad77dc509e0fb59c5b/image.png)

Story points: 6.0
Experiment Detail: introduce dynamic data to exp. detail view

### Problem to Solve
In the experiment detail view there are for alpha 1 three tabs available: details, training log and files. The details view shows a top level information on the experiment execution. Much of the current data is static. 

This ticket aims at making the content dynamic for: 

**Metadata**
* [x]  Experiment name: Here the name of the experiment needs to be shown (big-dolphin_30012020 without the experiment/ )
* [x]  ID: This needs to be a UID. As an experiment is currently also a branch, lets use the UID from gitlabs branch. 
* [x]  Created: The time format seems complicated. Let's use this format: 30.01.2020 - 15:24:34
* [ ]  Training time: Taken from gitlab pipeline runner in the format: 07:32:15
* [x]  Completed: If still running write ---, if canceled write ---, if failed write ---, if completed, write start 30.01.2020 - 18:25:13
* [x]  Owner: user_name who started the experiment

**Source summary**
* [ ]  Data source: write and link the data source, either a branch or data instance. The link goes directly to the data instance detail view or the data repo view with the selected branch. 

**Model parameters**
* [ ]  List all standard and advanced parameters with the values and, if available, the description of the parameter.


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 4.0
Upload a file in data repo

### Problem to Solve
As a user i want to be able to upload a new file to my repository via an UI interface. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD: https://xd.adobe.com/spec/254df67a-f28e-4fff-5703-2a4183dde1e6-537c/

![upload_a_file_A](/uploads/1aea4bd6d86da8b54bdc360c50d20f0f/upload_a_file_A.png)

![upload_a_file_A-1](/uploads/dfeaee4cc24e9b97b56f1bf68257475b/upload_a_file_A-1.png)

![upload_a_file_A-2](/uploads/96cbc4cebfdd46bc06e9c797ab14c12f/upload_a_file_A-2.png)

Story points: 12.0
Backend: Redesign Experiment domain model

### Problem to Solve


There was an error/misunderstanding in the backend data model.

It is not: Run experiments which are pipelines on data, but run DataPipelines on DataRepository to DataInstance with cleansed Data.
Run Experiment (just choose files and Model) to create Experiment output

* Pipeline-DataInstance: DataRepository + DataPipeline-Configuration -> (DataInstance - Execution) -> DataInstance
* Adhoc-DataInstance: DataRepository + DataPipeline-Configuration -> (DataInstance - Execution) -> DataInstance
* DataRepository + DataInstance + Experiment-Configuration -> (Experiment - Execution) -> Experiment

An Experiment is both an Experiment-Pipeline-Configuration and its Experiment-Instance. The output can be a model an some Files

Experiment = ExperimentPipeline + ExperimentInstance
* has a name and generated slug
* has experiment.mlreef.yaml
* points to DataInstance or DataRepository
a) DataInstance
 * choose files
 * configuration is transient, configure Model directly
b) DataRepository+branch
 * choose files
 * configure Model directly, 
 * just info about files and branch name
 * no info about DataProcessors preProcessing


### Technical Solution

* [x] Refactor Experiment
* [x]  - must not include preprocessing pipeline
* [x]  - must point to a DataInstance
* [x]  - needs a name
* [x] think about filename experiment.mlreef.yml
* [x] remove dataOperations config

Story points: 4.0