giseldo/neodataset huggingface dataset:
Include lables for projects view (marketplace)

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
As a user I want to see if: 

a. A project is public of private

b. My role in this project (e.g. maintainer, owner, developer...)

c. If a code repository has been published.


### Intended users


### User experience goal
 

### Proposal for Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Permissions and Security


### Documentation
The design can be taken from the groups-projects view: https://xd.adobe.com/view/4cec0111-f89b-4825-6363-58d95e361fef-3add/


### Availability & Testing


### What does success look like, and how can we measure that?



Additional Notes
=====================
### What is the type of buyer?


### Is this a cross-stage feature?


### Links / references

Story points: 5.0
Upload file - 413 error message needs adaptation

### Observed behaviour

When uploading a too big file via the file uploader, the error message from Gitlab is: 

Failed to load resource: the server responded with a status of 413 (Request Entity Too Large)

The frontend (toaster) shows: "File could not be uploaded". 

### Expected behaviour

The frontend (toaster) error should show: "File(s) could not be uploaded. File limit of XX Mb exceeded."

- What is the current file limit?? 

### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 4.0
Error in Data instance Details view

### Observed behaviour
![Screenshot__68_](/uploads/6bbe327fdfbfcb15a2e810b4739c2255/Screenshot__68_.png)


### Expected behaviour
The user can access data instance detail view without any bugs.


### Steps to reproduce
1. Go to the data instance details view
2. Copy the url.
3. Open a new tab
4. Paste the link in the new tab and the UI breaks.


### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 8.0
Show status of datasets in dataselector branch dropdown

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
As a user I want to see the status of a dataset that might be still running in the data selector, branch drobdown.

For example, I create a new dataops pipeline where the branch and the dataset are inmediately created. I should not be able to select this dataset for any pipeline, as the pipeline is not finished. The best approach would be to show the data set in the data selector dropdown, but it is not selectable and it has a view of the pipeline status. 


### Intended users


### User experience goal
 

### Proposal for Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Permissions and Security


### Documentation
- Red means: Pipeline failed or was cancelled.
- Orange means: Pipelines is pending or running.
- Green means: Pipeline means success and dataset is ready to be used. This is the only one that is selectable.


### Availability & Testing


### What does success look like, and how can we measure that?



Additional Notes
=====================
### What is the type of buyer?


### Is this a cross-stage feature?


### Links / references
XD Link: https://xd.adobe.com/view/f059d209-7ad7-4789-7f96-20b99b464f11-8668/

![image](/uploads/adb41a5197352e41ca370248e1133b0b/image.png)

Story points: 7.0
In the jobs view, the filter button "finished" does not show finished jobs

### Observed behaviour

In the jobs view (insights tab), the filter button "finished" does not show any jobs that are finished. 

Finished are jobs that are either
- success
- failed
- cancelled

### Expected behaviour



### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================


relates to:

Story points: 3.0
Add systemTest for Publishing Docker Images to Gitlab/MLReef infrastructure

[MLReef&#39;s Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
The most basic part of publishing user-generated Data Processors within MLReef is building and pushing 
Docker Images to MLReef&#39;s Gitlab based Docker Registry.

It might be necessary to activate or configure the Docker Registry -> https://docs.gitlab.com/ee/administration/packages/container_registry.html#enable-the-container-registry

There exists already a [test project on gitlab.com](https://gitlab.com/mlreef/devops/develop-data-processor) which can be cloned and then pushed to the relevant branch environment or other infrastructure.

### Todo

* call the API codeProject
* checkout codeProject
* add some mlreef.yml manually to the project
* push it
* test it? 
  * does a pipeline run?
  * does it succeed?
  * API call to check docker registry if something is there?

### User experience goal
MLReef Developers can manually define a Dockerfile and mlreef.yml file to build and push docker images to the MLReef Docker registry.


### Proposal for Technical Solution
The smallest and simplest possible Dockerfile as a reference to `alpine:latest` without further modifications
```Dockerfile
FROM alpine:latest
```

The corresponding `.mlreef.yml` would look something like this:
```yaml
build:
  image: docker:19.03.5
  services:
    - docker:dind
  script: |
    echo "$CI_REGISTRY_PASSWORD" | docker login "$CI_REGISTRY" --username="$CI_REGISTRY_USER" --password-stdin
    docker build --tag "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" -f Dockerfile .
    docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
```

For more references of how to build docker images with Gitlab&#39;s CI, you can look here
* https://gitlab.com/mlreef/mlreef/-/blob/develop/web/.gitlab-ci.yml
* https://gitlab.com/systemkern/s5/-/blob/master/.gitlab-ci.yml


### Permissions and Security


### Documentation
The implementation of this ticket will yield much clarity about how a future MLReef Dockerfile template has to look like in order to be useful and usable


### Availability & Testing
This is a core feature and testing for this has to be completely automated. While writing the testing code, keep in mind that we will support on-premise installations.


### What does success look like, and how can we measure that?
MLReef Developers can build and push docker images to the MLReef Docker registry.



Additional Notes
=====================
Keep in mind that, for on-premise installations, we will need to be able to provide our base images preinstalled with the installation.

Story points: 2.0
Call the backend star project function

### Problem to Solve
From the frontend, call the backend endpoint to star a project when the star button is pressed.
![image](/uploads/303a73d9e9dc7254becf4ae4f487b7cb/image.png)

### Technical Solution
Change url to address the call to the another API


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 3.0
Backend: REST is docs incomplete

### Observed behaviour

https://mlreef.gitlab.io/mlreef/rest-api/feature/fix-system-start/index.html
Half of rest-docs is missing:

```
GET /projects
Example request
Snippet curl-request not found for operation::genericprojects-retrieve-all

Response body
Snippet response-body not found for operation::genericprojects-retrieve-all

Response fields
Snippet response-fields not found for operation::genericprojects-retrieve-all

GET /projects/public
Example request
Snippet curl-request not found for operation::genericprojects-retrieve-public

Response body
Snippet response-body not found for operation::genericprojects-retrieve-public

Response fields
Snippet response-fields not found for operation::genericprojects-retrieve-public
```


### Solution

* [ ]  Fix or remove those lines, some of those "projects", "generic projects", etc, are duplicates maybe

Story points: 2.0
The joblog views (e.g. experiments / insights) need to refresh automatically

### Problem to Solve
The job log view should automatically refresh - at least as long as it is in "running" or "pending" stage. This makes it easier for the user to follow the progress. 


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 5.0
Re-run a pipeline the same way it was initially executed

### Problem to Solve
As a user I want to simply re-run a past experiment, dataops pipeline or data visualization. This is done by clicking on a "view pipeline" button in the resulting experiment / dataset / datavisualization which brings me back to the pipeline view, with **all** parameters set as they initially was. This includes: 

- Data path from data selector selected and pointing to the state of the data as it was (commit history)
- All data operators are opened with the hyperparameter values set as they were
- The data operators' script needs to be the same as it originally was. This might be more tricky, as we somehow would need to store the version of the script used. In relation to our publishing process and the docker images, this might be a challenge...


### Technical Solution
Q: How do we solve the script versioning in relation to the publishing process?

**Detailed implementation: **

The commit id should be received in the view in order to rebuild the same project files structure when the data operation was executed. This involves only the file selector, the Gitlab API request should send the commit id as the ref, and then the user will see everything as it was then.

![image](/uploads/b28361b4c3166878249b933fc345452c/image.png)

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 6.0
Link data versions in pipelines

### Problem to Solve
As a user I want to be able to easily track what version of files I used for any of our pipelines. For example, in the experiment executive summary view this is done by clicking on the data links (see additional notes).

### Technical Solution
Use GIT (commit history) to track the data used:

MLReef Pipeline -> Gitlab Pipeline -> Commit Hash

For example: 
"master" should link to "185b6788dd948bc9c93be5ee01081dd3640eec30 on master"

IDK if our frontend can handle this already but in Gitlab it would bring you to the state of the repo at this exact commit.

https://gitlab.com/mlreef/mlreef/-/tree/185b6788dd948bc9c93be5ee01081dd3640eec30


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================
![image](/uploads/107adbc1af17fd8483a7ac0d070996e8/image.png)

Story points: 4.0
Add sorting options to the market place view

### Problem to Solve
In the marketplace view (visitor) show to tabs: 

* "all" -> sort by most recent commit
* "popular" -> sort by most stars (4) (FE ticket pending)

![image__4_](/uploads/c10c14ce578792ec85d01c123ee5152e/image__4_.png)

### Technical Solution

* This information for example "most recent commit" is not currently available in the projects response payload. So for this first case a good solution would be to show all projects without sorting.

* The popular sorting option is actually doable.

### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Link to XD: https://xd.adobe.com/view/5d9dcfd2-3bfb-4ee0-5069-8c9e2f0a5b7b-2bac/

Story points: 4.0
Backend: Add user update endpoint and role information

### Problem to Solve
As a user I want to modify some information for the user:

- [x] 1. Store in backend that the user has accepted terms during registration(**true** or **false**).
- [x] 2. Store in backend and send to mailchimp that user want newsletters(**true** or **false**).
- [x] 3. Store in backend user type after registration, these are the next:
- [x] 4. Delete pre-Fix of users "mlreef-users-"username" in Gitlab

![image](/uploads/7e729ebb6f939331db48b87dda0a9604/image.png)

Story points: 3.0
Add "Starring" feature for Projects

### Problem to Solve
As a user I want to rate projects by giving them stars (similar how github / gitlab) does it. By doing so, i can then filter / order projects by stars in the marketplace view or in the pipeline views (e.g. experiment pipeline). 

Note: in addition to addings stars the user can also remove a previously given star to a project.


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 5.0
Backend: Protect data from inconsistency by creating constraints in DB

### Problem to Solve

As we have many methods like "findByGitlabId" or "findBySlug", we have soft business requirements that some entity fields are unique or "kind of" unique. There was an error, when gitlabId was used twice and everything broke down due to our security mechanisms.

Therefore we could do two things:
* replace those soft "unique"s with lists
* harden and use constraints

We have very weak constraints relations in Postgres. Sometimes we got inconsistency state in db that requires manual fix by developers. 


### Technical Solution
1) Create constraints in database
2) Check the code and possible add business logic that rejects inconsistent data before it has gone to database
3) if some "soft constraints" are not hard enough to be "unique", those methods must change to support lists!


### Terminology
Constraint - programming entity in databases that links tables among each other and does not allow to create records that do not satisfy this linkage

Story points: 4.0
Backend: The Forbidden error

### Observed behaviour

Since a couple of weeks I witnessed that some endpoints were throwing a 403 error.

At the beginning the only one was the experiments creation endpoint

![image](/uploads/3a3a23c1a47a5b2e6b79875bc7cf4efb/image.png)

but in the last days I noticed that more of them are throwing this error.

![image](/uploads/30394912eaddc32ad74b1b8f4d6a8031/image.png)

I even tested with Postman to discard the *FE* and got the same:

![image](/uploads/308a28d63572aaccc67528bc62b7764a/image.png)

My conclusion is that a global security problem was introduced in the code in the last weeks. So this ticket seems completely necessary.

### Expected behaviour

Endpoints work for authenticated fronend users

### Steps to reproduce

1. Login
2. Click on Models

or 

Elaborate a Postman request

Story points: 5.0
End2end Testing on all developer machines

During the implementation of 555 it became clear that backend and to an extend EPF developers need to be able to run automated round trip tests on their developer machines.

Currently we do not have a really good integration testing concept. Though not the main focus of this ticket, it makes sense to think about a sustainable test concept as well.

Some of the needed tests are:
* start a model training 
* upload training statistics to the Backend
* upload trained model results
* start data pipeline and verify results
* cancel experiment


## Additional Notes
There currently exist some bash and curl based tests which could be extended and adapted, but dont seem to run on each machine.
There exists also an existing integration test concept, which spins up a gitlab machine and has a clean, reproducible state to run tests. Integration tests can be run on each machine which could start the server, as they rely on docker and kotlin. There are already many integration tests for various reasons which tests different strategies and can be debugged, this concept seems better in terms of maintainability, system architecture and matches the rest of the technology.

On Linux and OSX these existing curl-bash-scripts tests already work pretty ok but @si-ge-st had problems running them on Windows.

Story points: 15.0
FE: Set the value of the parameter "input-path" with the path from Data selector

### Problem to Solve
“As a user i will choose the data input using the Data Selector, this path should be the value for the hyperparameter input-path”

Set the value of the hyperparameter called "input-path" with the Path from Data Selector. This hyperparameter can be overwritten by simply writing another input path over the "standard" value from the data selector.

For this, we need to: 
- [ ] Limit the number of selections a user can do in the data selector to 1 (either a folder or file)
- [ ] Redisign the data operators cards to include the automatic input-path from the data selector

### Technical Solution
Probably, the input-path field needs some automatic updating. For example, if I start by dragging data operators in the pipeline and then select data from the data selector, the input-path field needs to be automatically updated.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Changed data selector for selecting only one folder or file - XD: https://xd.adobe.com/view/cd08a4b8-ecab-4508-4e6a-a00bb3967697-bb39/

Changed input-path input field: https://xd.adobe.com/view/566ad5dc-edcf-49e3-7230-c4d0e1fa4078-205b/

Story points: 5.0
Budgeting Service

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
<!--
  AKA. Requirements
  Describe the desired state of the system in the style of:
  “As a (User / Manager / Admin / Guest) I want to be able to do x”

  Also describe here any ideas for splitting the implementation into multiple iterations.
--> 

### Intended users


### User experience goal
 

### Proposal for Technical Solution
<!--
  AKA. Specification
  Add technical implementation details and the results of the ticket's discussion here.
  Specifications written here need to be taken seriously.

  If you write speculation here, mark it as such.
-->


### Permissions and Security
<!-- e.g: Which permissions are needed -->


### Documentation


### Availability, Testing & Test Cases


### What does success look like, and how can we measure that?



Additional Notes
=====================
### What is the type of buyer?


### Is this a cross-stage feature?


### Links / references

/cc @rainerkern

Story points: 10.0
Billing Backend

[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)

### Problem to solve
The backend should track the following points for user / group billing: 
* Tracking of tier account of the user (free, tier1="pro")
* Tracking of time of upgrade of an account and expiration date (for auto-renew or auto-downgrade)
* Projects the user upgraded with "accelerated inference" package for boosting model show inference times
* Number of projects he has boosted (not sure yet, but possibly the "accelerated inference" package will allow the user to boost more than one project (e.g. 3 or so).
* Keep payment information
  * payment type and information? e.g. credit card credentials? // security?? 
  * transaction history
* Create email notifications for accepted upgrades, automatic renew, auto-downgrade.



### Intended users


### User experience goal
 

### Proposal for Technical Solution
<!--
  AKA. Specification
  Add technical implementation details and the results of the ticket's discussion here.
  Specifications written here need to be taken seriously.

  If you write speculation here, mark it as such.
-->


### Permissions and Security
<!-- e.g: Which permissions are needed -->


### Documentation


### Availability, Testing & Test Cases


### What does success look like, and how can we measure that?



Additional Notes
=====================
### What is the type of buyer?


### Is this a cross-stage feature?


### Links / references

Story points: 3.0
Review mobile responsiveness

### Observed behaviour
The MLReef app is currently not optimized for mobile views. We should optimize this on it1 for the most relevant screens: 

- [ ] Marketplace
- [ ] Repo overview
- [ ] Pipeline creation views
- [ ] Experiment overview


### Expected behaviour



### Steps to reproduce



### Terminology
> Define terms that are important in the context of this task



### Solution
> Add technical implementation details and the results of the tickets discussion here.

* [ ]  Add at least one test



Additional Notes
=====================
![3E68AB8F-FC74-45DD-865A-99400B610B32](/uploads/15d7862a03cb70392b12945c326569ac/3E68AB8F-FC74-45DD-865A-99400B610B32.png)

relates to:

Story points: 40.0
Publishin process - specification

### Problem to Solve
As discussed with @flohin we need to continue the final specification for the publishing process. I, @cpmlreef, would suggest to do this in 2 weeks time (next sprint)


### Technical Solution
> Add technical implementation details and the results of the ticket's discussion here.


### Terminology
[MLReef's Glossary](https://gitlab.com/mlreef/www-mlreef-com/-/tree/master/handbook/glossary.md)



Additional Notes
=====================

Story points: 5.0
Backend: Rename some data operators

### Observed behaviour

As user I have witnessed that some names of projects can be misleading. 

The patten applied make them look like a slug.

![image](/uploads/7e5586da7dcb3df819f1bb0667099c7c/image.png)


### Expected behaviour

Rename these autogenerated projects and further registers so that the name is more user friendly

### Steps to reproduce

1. Go to http://ec2-3-127-170-253.eu-central-1.compute.amazonaws.com/#personal
2. Login
3. Read names


### Terminology
Project names, slug

Story points: 1.0
Prioritize Cypress integration tests

### Problem to Solve
As develpers, we need to create cypress integration tests for the higher priority functionality within the user interface workflow. It is plausible to perform this operation in the form of iterations/steps. In the current scope of this ticket, we can focus on e2e tests for experiments and pipelines workflow.


### Technical Solution
Research and understand the e2e testing process, propose a mechanism and technology we need to launch this process. So that we can automate the process of UI testing.


### Terminology
Cypress, Integration tests



Additional Notes
=====================

Story points: 16.0