giseldo/neodataset huggingface dataset:
Implement a CAS - Proof of Concept

### Problem to Solve
We need to provide group membership and security authentication.


### Technical Solution
Use Gitlab as Central Authentication Service

**Permissions:** How to solve role- **AND** resource-based permissions with spring

eg: German is Owner of repo *Cats* but has only Guest privileges of repo *Dogs*

Simple logic can be implemented in Annotations with `SPEL` complicated logic can be implemented in e.g. Spring's RequestFilterChain or SecurityFilter. Another solution is to use aspect-oriented programming
**Data Access / Caching:** Token handling can be handled by caching the JWT tokens / the whole user Blob in Redis

**OAuth Tokens:** will be sent in the response body

* [ ]  How does the EPF authenticate against the backend?
       During the creation of Pipelines, the EPF receives a token from the backend. This token is scoped as narrow as possible for a single project or Gitlab pipeline id (or even experiment / model / …) 
* [ ]  Will environment variables persist between pipeline retries - [gitlab docu](https://docs.gitlab.com/ee/api/pipelines.html#retry-jobs-in-a-pipeline)
* [ ]  Should the Frontend create groups directly in Gitlab?


### Terminology
> Define terms that are important in the context of this task

Additional Notes
=====================
Additional info can be found in the [Epic](#389)

/cc @si-ge-st @flohin

Story points: 4.0
Several bugs in commit detail views

### Observed behaviour

In the commits detail view (inital commit), there are several bugs / missing infos

* [x]  ... authored 4 days ago (the timer seems mocked)
* [x]  Commit message has mocked text
* [x]  ... showing xyz files in xzy additions... the numbers in the initial commit is 0, despite having many new files

In another commit (from a pipeline by entering the branch view and selecting the commit code): 

* [ ]  the counters of xyz files changed shows changes, but these are not shown further down

### Expected behaviour



### Steps to reproduce

Open develop, open cats_dogs project, open commits view and select "initial commit" for the first bugs. 
Same procedure but select "branches" and there select the commit ID of any branch for the second bug.

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

Link to XD design: https://xd.adobe.com/view/489d424c-6bd9-41c5-43cd-0ad0f220adf4-3eac/

First bug list (from commits view): 
![image](/uploads/cb12f63580addcaaaa4858ceb5e32fa4/image.png)

Second bug list (from branches view):
![image](/uploads/902887ce8c215ca2cb5d568261d2ea9c/image.png)

Story points: 2.0
Deleting operators in pipelines view mixes up what it deletes

### Observed behaviour

In a pipeline (doesnt matter which), when I duplicate an operator with the duplication function and then, afterwards, delete the first operations (pressing the x button), the second (duplicated) operator gets deleted, not the first (where i clicked the delete button). 

### Expected behaviour



### Steps to reproduce

Develop environment, open up data pipeline, drag an operator to the list and write something in a para field to keep track of it, duplicate the operator, delete the first operator. The second will be deleted, not the first. 

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/428ef262cf648dbfb4f17467fbca144a/image.png)

Story points: 6.0
Some views were broken after style were merged

This is a list of errors caused by the last change in the styles, so everyone is encouraged to contribute with this list.

Every error (or suggestion) should be numbered. 

### Error reported status:

1. done
2. done
3. done
4. done
5. pending
6. done
7. done
8. done
9. done
10. pending
11. done
12. pending
13. done
14. done
15. done
16. done
17. done
## The list of errors:

1. ![image](/uploads/37f17d980f356b7a89e0abab5934871c/image.png)

2. Navbar should spread wide
![image](/uploads/ed193b3d25d2ea7787eb2da1c742af35/image.png)

3. Experiment card buttons and space
![image](/uploads/8d0ca8d3cc69929a4afe481494688017/image.png)

Story points: 2.0
Almost all automatic tests are broken

### Observed behavior

The automatic tests got broken after installing the new scss support libraries
![image](/uploads/f6ada36bc25e965b3e039ed211277958/image.png)

### Expected behavior

All tests should run correctly without errors

### Steps to reproduce

Execute `npm test`

### Possible Solution
> File extensions have to be included in the configuration of the tests.

Story points: 2.0
File view has some dummy content

### Observed behaviour

In the file view there is still some dummy content that we need to change to dynamic: 

- Commit date is set at 4 days instead of real time passed 
- Contributors is set to three without the avatars of the contributors

### Expected behaviour

Expeted to be dynamic. 

### Steps to reproduce

1. Open develop environment
2. Open a project with some files in the repositorory
3. Open a file withing the repository to access the file view 

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
How it should look like from XD file: https://xd.adobe.com/spec/808428ab-eb22-4924-5713-82a0e4109865-f6b5/

How it looks like:
![image](/uploads/469a8bb81d00f86006b97534fff04bf4/image.png)

Story points: 5.0
ML Project / settings / members

### Problem to Solve
As a user I want to be able to change who is member in a project. This is done through the settings / members view. 

Only the "maintainer" role can view this page. There are two different types of member invitations, individuals and groups. The main difference is, that when inviting an entire group, all group members get access with a defined maximum permission level. 

**Permissions:**
How to solve role- **AND** resource-based permissions with spring

eg: German is Owner of repo _Cats_ but has only Guest of repo _Dogs_

Simple logic can be implemented in Annotations with `SPEL` complicated logic can be implemented in e.g. Spring's RequestFilterChain or SecurityFilter. Another solution is to use aspect-oriented programming


### Technical Solution Backend


**Data Access / Caching:**
Token handling can be handled by caching the JWT tokens / the whole user Blob in Redis


### Terminology

*  Searchable members are all MLReef users. They can be within the group or external users. 
*  Inevitable groups can only be sub-groups within the project group. External groups will NOT appear in the search and can not be invited.

Additional Notes
=====================
Link to XD: https://xd.adobe.com/view/06099f51-b2c6-4532-4020-800b4f7e5efa-7038/

![Settings_Overview___2](/uploads/7ba6c330b030e93a9f9aa6830a8f1c26/Settings_Overview___2.png)

![Settings_Overview___1](/uploads/8110f6f2290589ec59882cac342805ad/Settings_Overview___1.png)

![Settings_Overview___3](/uploads/0f5493ff3bf1a39f2a0b66553b22e214/Settings_Overview___3.png)

![Settings_Overview](/uploads/c2be2dd626fa4eac24de28514350eb3c/Settings_Overview.png)

Story points: 12.0
Frontend: Use MLReef-Backend for Experiments

### Problem to Solve

The frontend has to use the Backend endpoints for Experiment manipulation

There are the following here: https://mlreef.gitlab.io/backend/feature/265-refactor-experiments/#_get_data_projects

```
Experiments - /data-projects/:id/experiments
GET …​/experiments
GET …​/experiments/:id
POST …​/experiments # create new one, update is not supported
GET …​/experiments/:id/metrics
PUT …​/experiments/:id/metrics  # just useful when called by EPF!
GET …​/experiments/:id/mlreef-file
POST …​/experiments/:id/start  # starts the execution of this Experiment
```

### Technical Solution

* [x]  Create a DataProject (https://mlreef.gitlab.io/backend/feature/265-refactor-experiments/#_post_data_projects)
* [x]  Use that DataProject to get the ID and the gitlab Project
* [x]  Create Experiment via that REST call
* [x]  Get the MLReef file (for debug purposes? ) 
* [x]  Start the experiment
* [ ]  Get the Metrics and update that chart in GUI
* [x]  Get the log output via gitlab rest

@flohin is here to help!

@SaathvikT 
See also frontend#265 and feel free to "order" more endpoints or more information you need!

Story points: 5.0
Find 2 - 3 data visualizations per data type

### Problem to Solve
For alpha 2 we need to have some data visualizations ready to use for our alpha users (availbale in the visualization pipeline). We need to have at least 2-3 data visualization per data type: 

**Images**
*  tsn-e (already available)
*  [PCA](https://scikit-learn.org/stable/auto_examples/decomposition/plot_pca_iris.html)


**Text**
*  [Wordcloud](https://github.com/amueller/word_cloud)

**Tabular**
*  [Histograms, Scatter Plots](https://towardsdatascience.com/introduction-to-data-visualization-in-python-89a54c97fbed)


### Technical Solution


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 4.0
Find 10 data processing operations for each data type

### Problem to Solve (The code is available in EPF, this story cannot be finalized)
For alpha 2 we want to have a set available for data processing for the alpha users to use. These operations need to rely on the top three data types, therefore we need to find (list) at least 10 data operations per data type and list them here (ideally with source code).

**Images** (Done)
*  Augment (already available contains 7 transformations in one script)
*  Random Crop : Randomly crops the image to a fixed size.
*  Lee filter
*  Color Jitter: Randomly change the brightness, contrast and saturation of an image.
*  Random affine: transformation of the image keeping center invariant.
*  Random Erasing: Randomly selects a rectangle region in an image and erases its pixels.
*  Add noise: Add different types of noise to images.
*  Image Rotation: Rotates the image.
*  Thumbnail: Creates thumbnail from image. 

(Done)
**Text (the concept is different here we cannot fabricate synthetic data with this transformations)**
* Bag of Words: Transforms the words to vector.(Done)
* TF-IDF: Calculates Term Frequency and Inverse Document Frequency of the words and documents.
* Stemming and lemmatization: The goal of both is to reduce inflectional forms and 
  sometimes derivationally related forms of a word to a common base form.
* Stopwords filtering: Remove the words that are present in all texts but don't provide meaning. Like articles, connectors, adverbs etc.(Done)
* Special Characters filtering: Some texts may contain special characters and that is not a desirable for training. (Done)
* Converting all letters to lower or upper case (Done)
* converting numbers into words or removing numbers (Done)
* removing punctuations, accent marks and other diacritics (Done)
* removing white spaces (Done)
* expanding abbreviations (not applicable)
* removing stop words, sparse terms, and particular words (Done)
* text canonicalization

**Tabular**
* Normalization: Normalize the data with mean and standard deviation.
* Filtering: Remove some data using some criteria, for example, "remove all rows with NULL values".
* Data Augmentation: Create random points with normal distribution parameters.
* Numerical to Categorical: Transform numbers to categories using ranges.
* Categorical to Numerical: Transform categories to a number.
* Data completion: Use the most frequent value of a feature to complete null values in other rows.
* Add Noise: Add white noise to a series of numbers (column), commonly used to avoid overfitting.






### Technical Solution
Code samples for image transformation
https://github.com/pytorch/vision/blob/master/torchvision/transforms/transforms.py
https://pytorch.org/docs/stable/torchvision/transforms.html

### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 8.0
Publish models and dataops in productions

### Problem to Solve
We want to offer a good amount of script samples to increase engagement from users in the page.


### Technical Solution

Publish models and datasets that are interesting for the users, in order to attract more attention to the site and also show the features of MLReef.
- [x] Datasets and models for image classification: Resnet, mobilenet, inception, vgg, etc.
- [x] Chatbot and text dataops
- [ ] GAN's, photo2cartoon (this one requires GPU)
- [ ] Music tagging
- [ ] Sentiment Analysis with BERT

* Age prediction -- too heavy to be used now but the scripts are tested

* Demand Forecasting  --require fbprophet still not installed due to image size concerns

* Sentiment Analysis -- requires language models still not installed due to image size concerns

### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Story points: 8.0
The select data div dissapers after selecting data in experiments

### Observed behaviour

In the experiments pipeline, after I select the data files through the modal, the data block dissapers from the data pipeline: 

![image](/uploads/3b9827cfa716ac1de5b96e254b3a2e2f/image.png)

**Note**: This is only in the experiment pipeline. For the other pipelines (data visualization, data processing) this works perfectly fine. 

### Expected behaviour

It should show this: 

![image](/uploads/cd9518c958401192a3978c532eb3b2b0/image.png)

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 1.0
Avoid complete new page load with spinner for data repo overview

### Observed behaviour

When opening the data tab coming from e.g. the experiments tab, the page goes whit with the spinner on the top left screen. This really feels odd. 

### Expected behaviour

The same handling as coming from the data tab to open the experiment tab - a smooth transition is expected. 

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 2.0
User hint appears two times for data visualizations

### Observed behaviour

The user hint appears twice when opening data visualizations: 

![image](/uploads/73ac8a163fd6d6a65a111b1ed350b5bd/image.png)


### Expected behaviour

It should open onle once.

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 1.0
ML project / settings / general / advanced - Delete Project

### Problem to Solve

As a user I want to be able to adapt the settings of my repository. 

**Note**: The "settings" tab in any directory is **only** visible for members of that repo. 

**Note**: These settings view is for the **ML project** repositories.

### Technical Solution

Note: The remove project opens the already implemented modal of deleting a project which currently is placed in the projects overview. We need to move the modal from the project overview to the advanced settings.

![image](/uploads/6cb44a9ed95a46f067b68bba0e867827/image.png)

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Documentation: https://mlreef.gitlab.io/backend/develop/#_delete_code_projectsid

**Relevant designs in XD file**

*  Slide 3

Link to XD: https://xd.adobe.com/view/65e88d76-2c29-4503-47d1-3148049ed484-13d9/

![Settings_General_Advanced](/uploads/76fefa60d0a8ff931b1b2eacf3a5f049/Settings_General_Advanced.png)

Story points: 6.0
Research on git LFS

We need to know how GIT LFS works, in order to see if this is the ideal solution for MLReef. 

Here are some questions that need to be answered: 

* [x]  Can git LFS be used for data deduplication? -> Maybe
* [x]  Are there major limitations? (sizes, performance, usage, etc) -> Not as far as I can see
* [x]  What does the user (client) need to setup in order to work? -> Major git clients support it out of the box
* [x]  What do we (server / gitlab / infrastructure) need to setup? -> Gitlab supports LFS

Additional Notes
======================
* Atlassian Keynote regarding LFS: https://www.youtube.com/watch?v=006RUPVIP-c
* Git LFS Client with AWS S3 backend: https://github.com/jasonwhite/rudolfs

Story points: 1.0
ML Project / Settings / General / Naming

### Problem to Solve

As a User, I want to be able to change the _name_ and the _description_ settings of my repository. 

**Note**: The "settings" tab in any directory is **only** visible for members of that repo. 


**Note**: These settings view is for the **ML project** repositories.

### Technical Solution
* Project Name is saved in Backend & Gitlab -> Update via Backend
* Description is saved in Backend only -> Update via Backend
* Avatar -> Not in the scope of current milestone

* Tags & Slug -> Not in the scope of the current milestone.

https://mlreef.gitlab.io/backend/develop/#_put_data_projectsid
```
PUT /data-projects/:id
Example request
$ curl 'http://localhost:8080/api/v1/data-projects/3b3e4811-fae3-46cd-8e07-84e3f94b3655' -i -X PUT \
    -H 'Content-Type: application/json' \
    -H 'Accept: application/json' \
    -d '{
  "name" : "New Test project",
  "description" : "description"
}'
Response body
{
  "id" : "3b3e4811-fae3-46cd-8e07-84e3f94b3655",
  "slug" : "slug-1",
  "url" : "www.url.com",
  "owner_id" : "aaaa0000-0001-0000-0000-cccccccccccc",
  "gitlab_group" : "mlreef",
  "gitlab_project" : "New Test project",
  "gitlab_id" : 1,
  "experiments" : [ ]
}
```


Additional Notes
=====================

**Relevant designs in XD file**

*  Slide 1
*  Slide 2

Link to XD: https://xd.adobe.com/view/65e88d76-2c29-4503-47d1-3148049ed484-13d9/

![Settings_Overview](/uploads/d66e912bc44a73e6895d2c8db1808ae8/Settings_Overview.png)

![Settings_General_Naming](/uploads/98a1c8845211aa62763435e476aaa9a3/Settings_General_Naming.png)

Story points: 8.0
Create experiment UUID

### Problem to Solve

Currently, in the experiment detail view, the experiment ID is the experiment name. We need to change this to be a true UUID, which is also stored in our backend DB (@flohin). 



### Technical Solution

As we are currently working with branches (an experiment creates a branch where it stores the output files), we could use the UUID created by the Gitlab for branches - if Gitlab does that? 

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================
> Give a more technical explanation of how you want the app to function. Include screenshots and links to the relevant documentation

Story points: 1.0
Refine CSS on experiments overview

### Observed behaviour

The current experiment overview does not look like it should. 

![image](/uploads/f05a07cf69609c2989df7f9adee3e624/image.png)

### Expected behaviour

It should look like this, link to XD: https://xd.adobe.com/spec/7d7e3c44-50a2-4980-5c95-c30af600762d-0b4f/

### Steps to reproduce

Current state of experiment overview. 


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 2.0
Experiment values not showing up

### Observed behaviour

A successfull experiment does **not** show the output values as shown in the attached image. 

### Expected behaviour



### Steps to reproduce

Open dev environment, cats dogs project, experiment tab and expand the experiment "super-sirene"

### Terminology
> Define terms that are important in the context of this task


### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================

![image](/uploads/1eaabaf0b62ce501130db6b995c95cac/image.png)

relates to:

Story points: 4.0
Forking in progress view

### Problem to Solve

We need an forking project view, as the forking progress might take some time. There are some "specialities" for this view: 

1.  There is a gif ;) 
2.  The top area needs to be hidden away (the grey area contracts and only shows what is included in the screen), because: 
3.  We need to automatically refresh the page to see if the project has been completely forked. Gitlab refreshes every 2 seconds or so. 


### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/b4d538d5-666e-40d6-614c-6a995806ee4d-c2e4/

![forking](/uploads/0a02eeaa043309e0ad74ab2da3e50513/forking.png)

![Forking](/uploads/0243c38f03da263c37a8c2d28fdda8a1/Forking.png)

Story points: 2.0
Empty experiments view

### Problem to Solve

We needed a nicer and more inviting view for empty experiments. This is it! 

Note: No filter buttons are present. The create a new experiment button now is centern and on the bottom. 

### Technical Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one unit test
* [ ]  If useful: Add documentation to the readme


### Terminology
> Define terms that are important in the context of this task


Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/e233e6ca-7ec7-4842-58ff-f724872b4612-9402/

![Experiments_empty](/uploads/d6da3b9c958c9dffa93bb3b41a0d92dc/Experiments_empty.png)

Story points: 1.0