giseldo/neodataset huggingface dataset:
Move end2end test code

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
Context: To make builds faster and more efficient, Gitlab allows adding conditionals to build scripts. So e.g. the frontend build and FE unit tests are only executed if frontend code has been changed.

e.g:
```
only:
  changes:
    - web/**/*
```

Currently, the end2end tests are located under `web/src/__end2end-tests__` and unfortunately, this means, whenever an end2end test is edited or added we have to wait for the frontend build (and deploy) to be finished.

So the topic of this ticket is to move the end2end sources out of `web/src/` so that we can use a Gitlab ci build filter on the `web/src` folder.



### Proposal for Technical Solution
Proposal: Refactor end2end test source to e.g. `web/test/__end2end-tests__` and `web/tests/__unit-tests__` 



Additional Notes
=====================
### Links / references
Frontend build takes about 18 Minutes

![build](/uploads/4af416ba0e8d57cc77d8524600e8528f/build.png)

Story points: 8.0
Change folder upload to file(s) upload

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour

The current function of uploading folders has several issues and need to be adapted to files upload: 

- [x] Drag and dropping a file into the drop field results in an error (it does simply not work). 
- [x] When opening the explorer view (by clicking on the drop field), the explorer want to find folders. We need to change this so the explorer searches for files to upload. Selecting multiple files (not only limted to one) should be possible. 
- [x] There seems to be a cap in size to upload of 10MBs in total (sum of all files). This is a limit that is not realistic, as there are no datasets that are below 10MBs. We should have a limit for this of severl GBs (if we need a limit). 
- [x] The root where the user navigated before in the tree view of the repository should be kept for uploading files. So if he moved to cats/train and presses the + button to upload files, the newly uploaded files should be placed there. 
- [x] This root information should also be shown in the upload view (if design is required, please tell). 
- [ ] The drag-drop function removes previous files, review that



### Expected behaviour



### Steps to reproduce
In staging, try the upload folder function. 

### Workaround


### Solution
<!-- Add technical implementation details and the results of the tickets discussion here. -->



Additional Notes
=====================

Story points: 13.0
Only a limited number of projects are shown as a visitor

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour

As a visitor I can only see a very limited amount of public ML projects when accessing mlreef.com/explore.

Furthermore, when logged in, my own projects now do not appear under "my projects" but only under the explore tab.


### Expected behaviour

I should see all public projects. 

### Steps to reproduce

Go to https://mlreef.com/explore

### Workaround


### Solution
<!-- Add technical implementation details and the results of the tickets discussion here. -->



Additional Notes
=====================

What one can see: 

![image](/uploads/c5ef8339b1916c72e84482d06e4799fa/image.png)

What one should see: 

![image](/uploads/199b7a2825d873a16384d31e46040945/image.png)

Story points: 11.0
Use constants for permission levels

The following discussion from !843 should be addressed:

- [ ] @andres46 started a [discussion](https://gitlab.com/mlreef/mlreef/-/merge_requests/843#note_422219857):  (+1 comment)

    > Hey @diegovinie 
    > 
    > what about using constants for the access levels?
    > 
    > like developer = 30
    > visitor = 10
    > 
    > Just to have an idea of what these numbers represent.

Story points: 3.0
Adjustments and minor bugs to review in the pipelines process

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behavior

- [x] In the ticket https://gitlab.com/mlreef/mlreef/-/issues/683 some changes were requested for the "Experiments Overview". 
These changes have to be replicated wherever the data sets and pipelines information is presented.
- [x] In the dataset details, the breadcrumb does not work.
- [x] Make the abort button work for the data set details view. The button is shown sometimes when it makes no sense because the pipeline was already canceled or finished by itsel - the abort button is shown in the detail view all the time. It should be replaced by an "X" if the job is somehow complete and the entry can be deleted.
- [x] Data visualizations overview and data visualization details view have to look like data sets and data sets details view
- [ ] Counter of data visualizations in the repo overview and number of data visualization entries in data visualizations overview do not match
- [x] hide datavisualization prefix "data-visualization/" - so show only the name
- [ ] Change the format of the time stamp for all pipelines :) currently -15102020-1 to **- 15/10 14:30:01** 
- [x] Can we include the weight of the branch in the "use: " section? Does gitlab provide this info? If not, lets delete this sections.
- [x] The view pipeline button has no function 

![image](/uploads/5482fb0485c4592cbe051d3c7733dfdb/image.png)

![image](/uploads/c10561ee1dba1e00e882ee6261e53a45/image.png)

Story points: 5.0
Create pipeline output entry as soon as pipeline started

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behavior
As a user I want to have inmediate feedback as soon as I created any pipeline and I executed it (by pressing the "proceed to results" button in the modal view"). Today, and most of the times, a toaster appears with e.g. "experiment created" but no entry is found in the experiment overview. This is the same for all other pipelines. 

Then, after a while a new toaster appears with pipeline was created and then, the entry is available. 

This time difference results in confusion and bad UX.


### Expected behaviour

The entry should be created immediately. The pipeline can start afterwards if the machines are not available yet. 

### Steps to reproduce

In staging, execute any pipeline and jump directly to the results. No entry is there until, if it goes fast, a few seconds but sometimes it takes minutes! 

### Solution
In order for the backend to create a pipeline in Gitlab, some steps have to be performed
* [PipelineServiceL209](https://gitlab.com/mlreef/mlreef/-/blob/master/backend/mlreef-rest/src/main/kotlin/com/mlreef/rest/feature/pipeline/PipelineService.kt#L209) and following

One step is to look for or create the EPFBot user. -> Searching is done by select all users from Gitlab and iterating through them. --> `AuthService.CreateOrFindGitlabUser()` --> AuthServiceL263.

`AuthService.CreateOrFindGitlabUser()` is the first place to be changed to a more scalable approach.

We believe, optimising this function will solve the performance problem.

First approach to try out -> https://docs.gitlab.com/ee/api/search.html#scope-users



* [ ]  Add at least one test



Additional Notes
=====================

Story points: 6.0
Projectview throws exceptions when requesting project info

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour

When accessing data in a finalized dataset, the following error appears and no data is shown. 

### Expected behaviour

The user should be able to navigate the data tree and discover the files in the file view. 

### Steps to reproduce
Use this link and navigate into a folder or file in the data view at the bottom of the dataset detail view:
http://staging.mlreef.com/MLReef_1/flower-classifier/-/datasets/06b72641-3c32-4b24-abc1-0991cabd1d0c

### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
![image](/uploads/07d916063422ad64bded08ff66de6196/image.png)

* http://ec2-18-156-26-49.eu-central-1.compute.amazonaws.com:10080/MLReef_1/flower-classifier/commits/data-pipeline/profound-shark-29092020-1
* http://staging.mlreef.com/my-projects/fcb1f674-0351-4ee8-b08a-a3ffa096ed9e/data-pipeline%2Fprofound-shark-29092020-1/commits

Story points: 3.0
Deleting files does not work properly

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour
As a user I want to delete files in my repository. This function seems broken as it does not do anything. Furthermore, the style is still old (no greyed out and blurried background effect). 


### Expected behaviour
The file should be deleted and the user redirected to the repo view. 


### Steps to reproduce

In staging, explore an owned project and then navigate into a file. Try to delete the file. 

### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

Story points: 5.0
Adapt frontend for authenticated visitors

1. [x] Hide execute button

![image](/uploads/ac1524cd43aa79bbb4f6d962e294e9d3/image.png)

2. [x] Hide *New Experiment* and *delete* buttons

![image](/uploads/61c38acbae6e7f9fa331a3b86c0b38da/image.png)


3. [x] Hide **+** button

![image](/uploads/b4ec7829c7c11dbba81773309a7fb229/image.png)

4. [x] Hide *abort/delete* button

![image](/uploads/7170f4d4096a377eb089c7f04f6ec2a9/image.png)


5. [x] Hide *delete* button

![image](/uploads/d814dfd90cc70efa80b03ed148e56843/image.png)

6. [x] Hide *delete* button

![image](/uploads/e14ce2daa72697d778f9911ff5c6ca13/image.png)

7. [ ] Hide *delete* button

![image](/uploads/5bc7046a106813592e8cb592fc908466/image.png)

8. [x] Fix branch button
9. [x] Hide *new branch* and *delete* buttons

![image](/uploads/cb23792e12073ade24346fde2eb08312/image.png)

10. [x] Hide *New Merge Request* button (merge requests view)
11. [x] Hide action buttons (edit, close) in merge request view
12. [x] Hide merge button in merge request view

Story points: 8.0
File size changes not shown in merge request view

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour
All changed files have a NaNKb size added...


### Expected behaviour



### Steps to reproduce



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
![image](/uploads/e9b9d8bfb61158042eaef2b2f6a29a42/image.png)

Story points: 2.0
Adapt pipeline views to include global marker and input icons

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
As a user I want to be reminded of the connex between pipeline types (e.g. data visualization) and the data operators that are available in this pipeline type. 

- [ ] Therefore, the corresponding global marker makes sense in these views as well. 
- [x] In addition, the information about input data type from the data operators makes complete sense.
- [x] In addition, include the star count from the corresponding code repos in the card list (as in the desing)

### Intended users


### User experience goal
 

### Proposal for Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Permissions and Security


### Documentation


### Availability & Testing


### What does success look like, and how can we measure that?



Additional Notes
=====================
### What is the type of buyer?


### Is this a cross-stage feature?


### Links / references

![image](/uploads/3ea27aa25d6ef656c121e75a47af07ea/image.png)

Link to adapted design (example for experiment pipeline, but the global marker needs to be set in all three with their corresponding color): https://xd.adobe.com/view/8d73060c-174a-43e6-6a96-78861d008514-fb42/

Story points: 4.0
Fix REST docs: /projects/:id/users/:userId

We have some duplicated endpoints, which offer the same features, but with a different syntax:

POST /projects/:id/users/:userId
POST /projects/:id/users?userId..
POST /projects/:id/users with objects

There is no need for that, we should just decide for an endpoint, design a pattern, and work consistently in that area.

Decide for an architectural style and just establish it.

## Todo

* Fix Rest tests and create snippets
* or refactor endpoints and delete obsolete rest docs

Story points: 2.0
A small graphical bug in settings / members

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Observed behaviour

In a project / settings / members. When I click on a project member the dropdown to change his role (no dropdown is present at the moment), a frame with scroll opens, which should not happen...

### Expected behaviour



### Steps to reproduce



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
![image](/uploads/3b6f35c851312dbc07b07c0e0a10a51b/image.png)

Story points: 2.0
Visual errors when accessing code repos from a pipeline

### Observed behaviour

I am in a pipeline (does not matter which) and click on a card to jump to the code repository: 

- [x] the green marker (ml projects) is applied to the code repository (instead of the code repo specific color)
- [x] the tab "data" is there instead of "code"
- [x] Had an issue with the data in the repo as well. In the code repository, the data from the data repo (e.g. training-data) was still shown. 

After refreshing the page the errors are gone and the code repo is shown properly. 

### Expected behavior
The global marker from the code repository should be applied. 


### Steps to reproduce

In the prod environment, open a pipeline, expand on any card and click "view repository"

### Terminology
> Define terms that are important in the context of this task


### Solution
For the solution, we can show the spinner while the project is loading or showing a small spinner for the project view comp, discuss this with @cpmlreef before the ticket's implementation.



Additional Notes
=====================
This is a data ops repo, colors should be Violette:
![image](/uploads/8b9b30aaabd16c96780f2c25c0e62203/image.png)

relates to:

Story points: 5.0
Job Log throws Error when job is still pending

### Observed behaviour
Sometimes it can happen that Jobs in Gitlab are pending for a long time. If for example there are no runners currently available.

In this case, the job log will throw an error (see screenshot)


### Expected behaviour
Since an empty log is expected for a pending job, some sort of icon or text which lets the user know that everything is ok


### Steps to reproduce
* Create a new Experiment pipeline
* Go to the job log fast enough so that you see a pending job


### Solution
> Add technical implementation details and the results of the tickets discussion here.




Additional Notes
=====================
![image](/uploads/2759eabd66db7a81c16a6722693f5af4/image.png)

Story points: 2.0
Implement modal for creating folders (directory)

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
As a user, I want to have a modal like Gitlab's one, in which the user can create a folder into a specific route. The route is selected, by navigating in the repository overview in the specific folder and then selecting the + sign and the menu option "New  directory".

### Intended users
All types of users with writing access.

### User experience goal
 
Ease the content creation in the repository.

### What does success look like, and how can we measure that?

### Design

Link to XD: https://xd.adobe.com/view/38fd22ad-6ab0-443b-afd9-89ba7cebcf1a-6dae/

![Create_new_folder](/uploads/a21100d273c1bae935a27cd0319c101d/Create_new_folder.png)

Story points: 6.0
Implement first stage of the publishing wizard

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve

Implement the user flow of the publishing wizard, the wireframes are ready and HTML elements will be done soon. 

Use these elements to create the UI from the "Select entry point view" to the interlude. This will be the first stage, it encompasses to provide all the files and data necessary to publish the code project and persist it in the database, call the API's and show the interlude view.

In the next link, the prototype can be found. For this issue *only* the slides 1-4 are required.

### Proposal for Technical Solution
These are the technical requirements for this stage: 

* Select entry point:

1. Publish one branch at the time which will be the master branch (protected branch from gitlab) as a starting phase. The commit hash is NECESSARY as well.
1. The entry point is the file or script to run in python.
1. The purpose of the publication process is to publish a data operator.

note: Requirement.txt should be placed in the root, and we try to find it there. Should not be picked by the user, validated in this stage.

* Select base environment:

1. Can only select one environment, only some that are very useful will be as options for the moment.
1. We need from backend the list of environments available, and an endpoint to send the environment selected(to be called after the publish button is pressed)

* Publish model:

1. Clarify the model type, whether more options should be shown or not.
1. An endpoint to list the model-types
1. An endpoint to list ml categories
1. The model type and ML categories selected should be a field in the backend and included in the publish endpoint
1. Boolean to persist whether the user accepted the terms of publication, manage versions(probably a timestamp).


### Links / references

Link to XD design: https://xd.adobe.com/view/2154ca1c-fbf7-4ba8-bac4-d5edc077d0ac-7407/

Story points: 15.0
CodeProjectPublishing API and Backend


### Problem to solve

The frontend developers will develop the publishing wizard with the mockups/wireframes.

During that development, we might find out the logical point when the frontend wants to send data to the backend (start the wizard) and what information the frontend sends and expects from the backend.

Due to missing planning, we dont know that yet in the backend. Therefore we want to try the upstream design approach: 

**-> Frontend tells Backend how the endpoint should be like, and backend follows that proposal.**

### Proposal for Technical Solution

we might need several endpoints:

* starting the publishing (backend checks project and request for validity)
  > the first iteration is very basic still. When An existing "unpublished" Data Repo is published, the following things will happen
   - [x] A `Dockerfile` is added to the project&#39;s Git Repo
   - Does the Dockerfile need to be customized?
   - [x] A `.mlreef.yml` is added to the Git Repo
   - does the `.mlreef.yml` has to be customized?
   - push the docker image as `:latest` version to the docker registry
   - [x] Gitlab triggers a pipeline based on the `.mlreef.yml` for every commit on the "master" branch
   - [x] this pipeline creates a new version of the Docker image
   - [x] pipeline is tested in end2end tests? --> based on !806 
   - [x] the new docker image version is published to Gitabl&#39;s docker registry
   - [x] parse the python file -> #780 
   - [x] persist the DataProcessorVersion from the Python file to the database on the first build
   - [x] update/replace the DataProcessorVersion on future builds (remember every commit to master triggers a new build 
* [x] fetch info or state of tasks/pipelines? -> go directly to GitLab
* [x] commit/cancel publishing? -> frontend can stop and cancel pipelines
* [x] stop publishing -> frontend offers to delete the `.mlreef.yml`
* [x] Publishing Porcess needs to store ./#{#MAIN_SCRIPT}" -->CORRECTLY <-- in database
* [x] Create an E2E TEST which tests a custom data operation -> [template e2e test](https://gitlab.com/-/ide/project/mlreef/mlreef/tree/master/-/web/src/__end2end-tests__/DataPipelines.test.jsx/); [target file](https://gitlab.com/-/ide/project/mlreef/mlreef/tree/master/-/web/src/__end2end-tests__/CodePublishing.test.jsx/)
* [x] Delete CMD from [Dockerfile template](https://gitlab.com/-/ide/project/mlreef/mlreef/tree/master/-/backend/mlreef-rest/src/main/resources/code-publishing-dockerfile-template/)

If everything goes as planned, end2end testing can be implemented in NodeJs with help from the FE team :) (see !806)

Story points: 2.0
Creating a group does not work

### Observed behaviour

In dev (and all other environments) creating a group does not work - probably because an image was selected? 

### Expected behaviour

The group should be built.

### Steps to reproduce

Create a group in dev with all fields filled up (in this case also as a public group)


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/045df4166e2a9b59a410e05239f252ca/image.png)
relates to:

Story points: 4.0
Backend: Cleanup project endpoints and fix visibilities

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve


**IMPORTANT: endpoint meanings:**

* own: I created
* my : my own and shared with me (member of project or group)
* public: everything public, can include own or my if they are public
* "/" all: my+public = every project I can use or attend
* starred: every project under "all" which has my star

Endpoints were mixed up, 

- add systemTests for OWN, SHARED, ALL and PUBLIC Projects
- add fixes to controllers
- add some more JPA and service methods
- add new unpaged endpoint

Story points: 2.0
Create UI component for Publishing Process

This are the list of required UI components for publish process' design.

Weight between bracets.

+ page 1
- [x] Tab Steps component [2]
- [x] Statement component [1]
- [x] Folder view component (from similar) [3]
+ page 2
- [x] Brick grid component (from similar) [2]
- [ ] Card component [1]
- [x] Filters component (from similar) [3]
+ page 3
- [x] Vertical Steps component [2]
- [x] Colored radio-button [2]
- [ ] Checkbox input (from similar) [1]
- [x] Vertical Tabs adaptation (few changes) [0]
+ page 5
- [x] Pipeline [5]
+ page 6
- [ ] Job display from (similar) [1]
+ page 7, 8, 9
- [x] Parameters display (from similar) [1]

Related to: 

## Resources:
https://xd.adobe.com/view/4dc536f4-bea0-453c-8826-406ae2f8312d-1896/screen/16412dee-042e-49e3-b502-d8719ab89f78/

## Notes:
- UI components don't perform API request nor Redux actions, therefore this issue doesn't encompass the whole publishing process rather the look and feel.

Story points: 24.0