giseldo/neodataset huggingface dataset:
ML :  Model for Demand forecasting

### Problem to Solve
> “As a user i can train a demand forecasting model following the instructions in the documentation provided by MLReef.”


### Subtasks
- [x] Find a dataset for demand forecasting that is widely used in ML platforms
- [x] Find a model that can be trained with this dataset and "port" to MLReef.
- [x] Add a README file for this dataset and model, it should contain an abstract and instructions to run the experiment.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

https://www.kaggle.com/tandonarpit6/store-item-demand-forecasting-challenge-fbprophet

https://github.com/LevinJ/Supply-demand-forecasting

https://github.com/antoniopaisfernandes/Demand-Forecasting

https://www.kaggle.com/c/demand-forecasting-kernels-only

Story points: 5.0
ML : Model for Sentiment analysis contains dataset , model , documentation

### Problem to Solve
> Describe the desired state of the system in the style of:
> “As a user i can train a sentiment analysis model following the instructions in the documentation provided by MLReef.”


### Subtasks
- [x] Find a dataset for sentiment analysis that is widely used in ML platforms
- [x] Find a model that can be trained with this dataset and "port" to MLReef.
- [x] Add a README file for this dataset and model, it should contain an abstract and instructions to run the experiment.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
https://github.com/vonsovsky/bert-sentiment

Story points: 5.0
Migrate Backend Documentation Publish to Main Repo

### Problem to Solve
Currently, in the main MLReef repo the backend rest docs are not published to Gitlab pages.


### Technical Solution

**Option 1**

The Gitlab pages for the main MLReef repo are linked docu.mlreef.com so the backend docu has to be rendered in a subfolder.

This was already implemented in the Backend repo and the backend implementation can be used as template

https://gitlab.com/mlreef/backend/-/blob/develop/.gitlab-ci.yml#L63

**Option 2**
Publish Restdocs on our own NGINX Gateway


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
* `backend/bin/ci-build-backend`
* `backend/bin/ci-unit-test-backend` (Rest Docs generation)
* `backend/.gitlab-ci.yml`
* `backend/Dockerfile` 
* `docs/.gitlab-ci.yml` (Pages Publishing) docs.mlreef.com

Story points: 5.0
Bug on pipeline creation (docker deamon)

### Observed behaviour



### Expected behaviour

We are recieving error messages on all pipelines: http://ec2-3-127-170-253.eu-central-1.compute.amazonaws.com:10080/mlreef/cats-and-dogs/-/jobs/4

### Steps to reproduce

dev environment and run any pipeline. 


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test

![image](/uploads/852ba4ddfefb6cb5d18888af992eee09/image.png)

Additional Notes
=====================


relates to:

Story points: 18.0
Sustainably fix URL workaround

### Problem to Solve
In order to fix a cross-cutting connection problem between EPF and Backend @rainerkern introduced a workaround script which would change environment variables in this EPF MR [epf!58](https://gitlab.com/mlreef/epf/-/merge_requests/58/diffs)


### Technical Solution
- [ ] use the actual CI SERVER URL in the `.mlreef.yml` in the backend
- [ ] remove the whole _environment_ script or at least the then deprecated parts of it from the EPF
- [ ] activate or delete this code line (https://gitlab.com/mlreef/backend/-/merge_requests/89/diffs#b191ad777ddb8beb02d92383f761d6ea89669730_59_54)
- [ ] activate or delete this code line 
(https://gitlab.com/mlreef/backend/-/merge_requests/89/diffs#b191ad777ddb8beb02d92383f761d6ea89669730_43_39)


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 20.0
Pipelines fail from wrong external URL

### Observed behaviour
The pipelines fail because the EXTERNAL_URL is corrupted and replaced with a random ID

![fail](/uploads/721835d4d10b8c7dbd8b38523f3a6e2a/fail.PNG)

### Expected behaviour
Pipelines work successfully


### Steps to reproduce
1. Create a data processing pipeline (ideally with Augment operation)
2. Check the job log for that particular pipeline.


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 8.0
Add support for tags when creating a project

### Problem to Solve

As a user, I want to add tags to the project that I will create. This will help to better describe the project and filter/index them in the future.

* FE should receive the list of tags in the Enum
* The project creation payload should contain the Tags field(array)
* The user should be able to adapt the tags later in the settings menu

Related to: backend!88

### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.
1. Fetch tag list from mlreef backend, [doc here](https://mlreef.gitlab.io/backend/feature/598-add-projects-endpoints/#_get_exploretags).
2. Create an auto-complete input using the fetched tags.
3. Include the in the POST following the [API doc](https://mlreef.gitlab.io/backend/feature/598-add-projects-endpoints/#_post_data_projects)

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
Following up https://gitlab.com/mlreef/backend/-/merge_requests/88#note_370855061

Story points: 5.0
Refactor data operation views

### Problem to Solve
> We currently have 3 views that behave very similarly; new experiment, new pipeline and new data visualization.

### Technical Solution
>  Since we can reuse more code than we currently do, create an only file which replaces just the data model depending on the view requested. This will lead to a more maintainable component instead of 3 files that share 90% of code.


### Terminology
Data pipelines, data visualizations, experiments

Story points: 5.0
Get experiment status from Gitlab

### Problem to Solve
We can currently cancel a running experiment through Giltab pipelines API, but since Gitlab and BE are not synchronized the Backend experiment keeps the status as RUNNING.
This situation causes confusion and frustrations to the user and developer because we show misleading info in the frontend. 

### Technical Solution

*Some Possible solutions:*
- Implement an update experiment endpoint so that we can sync both systems
  - builds on the existing cancel experiment endpoint
  - still need for handing failed pipelines; since it is possible that they do not report their status

- Get the experiment Status from Gitlab
  - Either: the Client does this (backend does not contain this information at all)
  - Or: Experiment Status is updated frequently **from** Gitlab (maybe even on every Request)
  - handling failed pipelines would be easier

Story points: 3.0
Adapt the view data pipeline button

### Problem to Solve

When migrating to the backend the Mlreef file was deprecated, so the "View pipeline" button has to be adapted. 
The button should adress to the "new pipeline view". The data operations and data input that the user selected for that operation has to be shown.

![image](/uploads/6e2586ae1c382a2c68b7a70eb609cfdd/image.png)

### Technical Solution

Get from the backend endpoint the information that was previously parsed from the mlref file.

Story points: 5.0
Complete view for User_Account / Settings / Profile

### Problem to Solve
As a user, I want to be able to change/update the username, avatar.

Gitlab requires administrator access to update such details, so it has to be implemented first in the backend

### Technical Solution
Please use backend for this ticket's functionality.

### Additional Notes

Link to XD: https://xd.adobe.com/view/811be39f-34d5-47ea-6bcc-95409cf5f74e-f72a/

relates to: #512 

**Already implemented**:
Status: Gitlab API

**TODO**:
Avatar, Username: Backend API

![Settings_Profile](/uploads/2183264d484e50e92d43b876da3bcabc/Settings_Profile.png)
=====================

Story points: 5.0
Backend: Allow groups to be added to project

### Problem to Solve
+ As a User that owns a project I should be able to add groups to my project.

+ Currently is only allowed to add users.

+ The ideal situation is **to invite** groups instead of just add them.


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.
Same treatment as users.

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

related: #389


Additional Notes
=====================

Story points: 4.0
Follow-up from "Create user settings page" - Refactor frontend routing

The following discussion from !512 should be addressed:

- [ ] @diegovinie started a [discussion](https://gitlab.com/mlreef/mlreef/-/merge_requests/512#note_364184452):  (+1 comment)

    > Hey @SaathvikT 
    > 
    > I really like this pattern for the routes, do you think we could implement it in the project routes?
    > 
    > cc: @andres46 

A better pattern for routes handling was found by Saathvik. It makes sense to apply it and refactor the router component in order to improve the readability of that code.

To apply this pattern in the routes file, an array of objects would probably be better. The object needs a field to indicate that the route is private or not. 
```
{"projectName": { "isPrivate": true, pattern: 'my-projects/project-slug/some-sort-of-name/param/:some-param' }}
```

So when we want to use it in a component, we can just replace :some-param in the code and use the link from a centrilized source.

Story points: 8.0
Backend: Project owner's uuid is not the same owner uuid

### Observed behaviour
Project owner's uuid is not the same owner uuid


### Expected behaviour
They should be the same uuid.


### Steps to reproduce

![image](/uploads/c58f93291596954fb19994323c848aa1/image.png)
Here look at the **ownerId**

![image](/uploads/fad2fe38c985e503d073017dbb873e51/image.png)
Here the user's id (in fact uuid) from the `/data-projects/:projectUUID/users` endpoint


### Todo

* [ ] add a flyway migration to add some more FKs
  * e.g., more subject-related FKs
* [ ] inform *all devs* when this MR is merged instantly
* *every machine with data inconsistencies will break and crash at beginning!*
* then we know, which machine has consistency problems and needs to clean its database state (dropping or fixing)

Please apply that patch LOCALLY:

```
-- If some strange data exists (Account is owner instead of Subject), this should fix it OR tell you that your database is broken
-- IF your local database is broken, please clean it. FKs and constraints already exist.
UPDATE mlreef_project
SET owner_id = a.person_id
FROM account a
WHERE owner_id = a.id;

UPDATE data_processor
SET author_id = a.person_id
FROM account a
WHERE author_id = a.id;

```


relates to: mlreef!503

Story points: 5.0
Create mlreef routes

### Problem to Solve
Create mlreef routes as compared to gitlab in ticket #441 

### MLReef public routes

```
https://mlreef.com/explore/mlprojects
https://mlreef.com/explore/visualizations
https://mlreef.com/explore/dataops
https://mlreef.com/explore/groups
https://mlreef.com/:username
https://mlreef.com/:namespace/:project_slug/
https://mlreef.com/:namespace/:project_slug/-/tree/:branch
https://mlreef.com/:namespace/:project_slug/-/blob/:branch/path/to/file/:file_name
https://mlreef.com/:namespace/:project_slug/-/blame/:branch/path/to/file/:file_name
https://mlreef.com/:namespace/:project_slug/-/commits/:branch
https://mlreef.com/:namespace/:project_slug/-/commits/:branch/path/to/file/:file_name
https://mlreef.com/:namespace/:project_slug/-/project_members
https://mlreef.com/:namespace/:project_slug/-/jobs
https://mlreef.com/:namespace/:project_slug/-/jobs/:job_id
https://mlreef.com/:namespace/:project_slug/-/experiments
https://mlreef.com/:namespace/:project_slug/-/experiments/:pipeline_ID
https://mlreef.com/:namespace/:project_slug/-/experiments/:experiment_ID
https://mlreef.com/:namespace/:project_slug/-/experiments/:experiment_ID#training
https://mlreef.com/:namespace/:project_slug/-/experiments/:experiment_ID#files
https://mlreef.com/:namespace/:project_slug/-/datasets
https://mlreef.com/:namespace/:project_slug/-/datasets/:pipeline_ID
https://mlreef.com/:namespace/:project_slug/-/datasets/:dataset_ID
https://mlreef.com/:namespace/:project_slug/-/visualization
https://mlreef.com/:namespace/:project_slug/-/visualization/:pipeline_ID
https://mlreef.com/:namespace/:project_slug/-/visualization/:visualization_ID
https://mlreef.com/:namespace/:project_slug/-/merge_requests
https://mlreef.com/:namespace/:project_slug/-/merge_requests/:mergerequest_ID
https://mlreef.com/:namespace/:project_slug/-/merge_requests/:mergerequest_ID/diffs
https://mlreef.com/:namespace/:project_slug/-/merge_requests/:mergerequest_ID/commits
```


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology

* **dashboard**: this is our marketplace. I like dashboard better as our marketplace is not a real marketplace where you buy stuff.


[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

related: #441

Story points: 0.0
Backend: Update user endpoint

### Problem to Solve
Initially we decided that User profile operations would go to Gitlab directly, but after some research there was found that Update operation was impossible by simple user (https://docs.gitlab.com/ee/api/users.html#user-modification). 
Therefore the update operation should be performed by backend using chain: frontend -> backend -> Gitlab


### Technical Solution
Add new endpoints for user update operation. Obtaining user information should go to Gitlab directly as it was decided before

* [ ] check, what data we want support in backend
* [ ] change name, email in backend
* [ ] solution 1) forward avatar image blob to gitlab
* solution 2) avatar will be changed via gitlab directly (no backend entity), if that is possible (which maybe it is not, then use backend for that)

### Terminology
Simple user - non-administrator user

* https://docs.gitlab.com/ee/api/users.html#user-modification
User update operation - changing name, email, avatar etc of user

Story points: 6.0
Esthetic bugs in develop branch

### Observed behaviour



### Expected behaviour
- [x] "New Merge Request" button in new merge request view has missing text:
![image](/uploads/1f4ec6f938780b284676f7f3162419f6/image.png)

- [x] Strange looking edges in the project avatar (with or without images, these edges appear)
![image](/uploads/924b4f71e46ed0721e19d09aec5f8f79/image.png)

- [x] Data visualization view has a grey line (should be deleted) and the filter buttons should be left aligned
![image](/uploads/9dbf9acd3a6a3cc4d722d8a86eb2abaa/image.png)

- [x] Same for datasets view: 
![image](/uploads/0a725abea01a22d6fd2fb85d9c17657e/image.png)

- [x] Select data modal has undifined as first rout? and the checkbox is hidden and not aligned with the file view:
![image](/uploads/4144c460dfa49222f4b192eda12c15c5/image.png)

- [x] The parameter name and the input field are not aligned properly (very slightly though)
![image](/uploads/965ef31376baf3960bbfe0754650e44e/image.png)

- [x] In the experiment list, the first entry (only the text on the left side) should be moved a bit down. And there is a separation line missing between experiments:
![image](/uploads/a19491164ca956ea684ea885c43839fb/image.png)

- [x] In the experiment detail view, the success status is red. It should be green on success, oragne on pending, green on running and red on failed or aborted. 
> please I need the step to reproduce it

**Open develop / open resnet50 project / go to experiments / open running experiment (holy-siren) / in the details tab the "running" is red**

![image](/uploads/6904ceeed37514ca5d4fc31a4e8a6dc8/image.png)


- [x] The download and delete buttons need mlreef styling ;) 
> please I need the step to reproduce it

**Open develop / open resnet50 project / go to experiments / open first successful experiment / move to files /**

![image](/uploads/c4849c390f83568efae8e763e95bb80c/image.png)

- [x] In the market place, when more than 3 projects appear, the filter and the new ML project buttons are separated. They should stay on the same when more projects are added. 
![image](/uploads/22a5173e45bc8a82446f2448f24bd3c2/image.png)

- [x] In the market place, the data types icons are always the same as well as the description. Probably also related to https://gitlab.com/mlreef/mlreef/-/issues/401

Link to XD: https://xd.adobe.com/view/5d9dcfd2-3bfb-4ee0-5069-8c9e2f0a5b7b-2bac/

![image](/uploads/1bce458db027c7c4a88fe68327375dcf/image.png)


### Steps to reproduce

All visual bugs are found in the current develop branch (12/06/2020)

### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 13.0
New Group leads to user profile

### Observed behaviour

When clicking on the navbar drobdown to "new group", one gets to the (still mocked?) profile view. 

![image](/uploads/3818eb3f99544b552ac706ea29d655f7/image.png)

@andres46 I think we already had the form for creating a new group, didnt we?

### Expected behaviour



### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 1.0
Extra parameters and fields required for data pipelines

### Problem to Solve
A data processing pipeline currently provides the following information:
![pipeline](/uploads/257372836cc04f6b9b164fd3dbad3514/pipeline.PNG)

Some additional information that is necessary from backend:
1. The corresponding gitlab pipeline information (as we have with experiments).
This will allow getting information such as **pipeline status**, **pipeline creation, updation times** and **duration**
of pipeline etc....

### Technical Solution
The required information is available for the frontend

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 1.0
Backend: Refactor DataProjects to have more fields from MarketplaceEntries

### Problem to Solve

* refactor Projects to have DataTypes[] (move them from Marketplace Entries)
* Tags are on Marketplace Entries

### Technical Solution


### Terminology

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

Story points: 5.0
Improve EPF Bot lifecycle together with group setup

### Problem to Solve

after mlreef#526 and backend!72 it is necessary to clean up technical debt
Problem: right now we use the user as GIT_PUSH_USER and GIT_PUSH_TOKEN, which is a security leak.
We already have te mechanism to create an "EPF-Bot" user, but now this was created for the user's group, and is a Maintainer, which is equally insecure as using the user, and even worse, is not really connected to project.

So actually we need to solve many problems:
* when creating a user, a user group must be created. *This group must be used as the owner* for every new project.
* For every dataproject of this group, a specific EPF-Bot can be created, for example "mlreef-project-$UUID-$ID-bot" for a data/code project with UUID $UUID and the connected gitlabId $ID.
* when a pipeline is started (before the gitlab pipeline call takes place), the following must be ensured:
  * the project EPF-Bot exists and has a token
  * that token can be stored as GIT_PUSH_TOKEN for that project (once per project is enough, no need for every CI)
  * that epf-bot username can be stored as GIT_PUSH_USER for that project 
  * the custom EPF_URL (to update/finish the experiment/pipeline) is created for this distinct PipelineInstance, therefore it shall exist as a pipeline-scoped CI variable and must not exist as a project-scoped CI variable

**Final Notice**: Right now we have the user as the owner of its resources. But actually we need to have the user's group to be the owner, to allow to add the epf-bot as a group member of that project (with DEVELOPER roles, should be sufficient). As a hack, we could add the epf-bot as a free floating user and just assign to that project. Maybe that is even easier, as we dont need the group-

### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 11.0
Remove SSLMate verification records from DNS Terraform env config + state

Noted https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1124, the import of records from DNS into terraform config obtained SSLMate verification records, which are maintained (via API) by SSLMate itself, and should not be in terraform.

They need to be removed from the config, and from the terraform state (so we don't actually delete them from Route53)

Story points: 1.0
Improve project restoration runbook

While working on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8239 I needed to spend a lot of time to research db restoration steps that are not covered in enough detail in the [runbook](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/community-project-restore.md).

We should add documentation for how to find the right base backup, which variables need to be set for the pipeline, necessary timeout settings to let the job complete, which gcp project the restore instance is running in etc.

Thankfully, part of that already is done in this branch: https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/tree/T4cC0re-master-patch-46363. But it's not merged and the README.md instructions contain typos (it should be `IS_RESTORE` instead of `IS_RECOVERY` and `PSQL_BACKUP_TIME` instead of `PQSL_BACKUP_TIME`).

/cc @T4cC0re

Story points: 3.0
Deploy redundant praefect nodes on gstg behind a load balancer

Spawning from a discussion on the Praefect readiness review https://gitlab.com/gitlab-com/gl-infra/readiness/merge_requests/10:

Let's try deploying redundant praefect nodes behind a GCP load balancer to avoid introducing a new SPOF in our infrastructure.

- [x] Terraform changes: Terraform changes https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1189
- [x] chef-repo MR: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2167

/cc @gl-gitaly

Story points: 5.0